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EDITORS’ WORD 





Dear Readers, 


Finally, we have a beautiful month of May. It’s 
warm, the sun is shining, plants and trees are 
blooming and we can enjoy all of that thanks to 
many public holidays. | hope that in your coun- 
tries this month brings many days off as well as it 
does in Poland and Denmark. If some of them are 
still ahead of you, | hope this issue will make your 
time off more pleasant. 


We will start with a very long “Adding ZFS to the 
FreeBSD Dual-Controller Storage Concept” arti- 
cle by Mikhail E. Zakharov. Grab a cup of coffee! 


The second article, “Secure VPNs with Gre and 
Strongswan for Small Business Networks with 
FreeBSD” by Antonio Francesco Gentile, will ex- 
plain how, with secure, reliable, and confidential 
communication, it is possible to reduce costs and 
improve the efficiency of production processes.. 


Now it is time for OpenBSD. Here you will read 
about “Reusing the OpenBSD arc4random in Mul- 
tithreaded User Space Programs” by Sudhi Herle. 
Upgrade your OpenBSD to the latest version and 
start your testing. 


Would you like to know “HOW to Install the XFCE 
4.12 Desktop on NetBSD 7”? Curt McIntosh will 
show you how! 


A lot of BSDs in this issue, huh? Well, here is the 
last one. It’s George Bungarzescus’ debut article 
about GhostBSD. Enjoy “FreeBSD Flavors. Do 
We Need Them? Today...GhostBSD. A (not too 
deep) journey to GhostBSD - desktop and enter- 
prise options - compared to pure FreeBSD”. 


In our interview, Fernando Rodriguez, Co-founder 
of KeepCoding tells us in the interview about his 
school, learning to program and that it pays off to 
be hard working. 


And in the end, as always, Rob Somerville and 
his comment on copyright infringement. 


Enjoy the whole issue and beautiful long days in 
May! 


Marta & BSD Team 
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bytes. And, profiling showed that reading from /dev/ 
urandom took a lot of time. | recalled that OpenBSD 
folks have a high quality userspace cryptographic ran- 
dom number generator called arc4random. But to 
the best of my knowledge, it was not generally avail- 
able for use in any Linux program. And so, it seemed 
like an excellent weekend project to create a portable 
version of the OpenBSD arc4random(3) generator for 
use in multi-threaded programs. 
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Baroness Neville-Rolfe DBE CMG, in the recently re- 
leased UK government document “Criminal Sanc- 
tions for Online Copyright Infringement” mandates a 
10 year prison sentence for serious instances of copy- 
right infringement. This intends to bring the penalties 
in line with those found guilty of copyright breaches in 
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to reduce piracy? 
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BSD Certification 


The BSD Certification Group Inc. 
(BSDCG) is a non-profit organization 
committed to creating and 
maintaining a global certification 
standard for system administration 
on BSD based operating systems. 


@ WHAT CERTIFICATIONS ARE AVAILABLE? 


BSDA: Entry-level certification suited for candidates 
with a general Unix background and at least six months of 
experience with BSD systems. 


BSDP: Advanced certification for senior system administrators 
with at least three years of experience on BSD systems. 
Successful BSDP candidates are able to demonstrate 

strong to expert skills in BSD Unix system administration. 


@ WHERE CANIGET CERTIFIED? 


We’re pleased to announce that after 7 months of 
negotiations and the work required to make the exam 
available in a computer based format, that the BSDA 
exam is now available at several hundred testing centers 
around the world. Paper based BSDA exams cost $75 USD. 
Computer based BSDA exams cost $150 USD. The price of 
the BSDP exams are yet to be determined. 


Payments are made through our registration website: 
https://register.bsdcertification.org//register/payment 


@& WHERE CAN | GET MORE INFORMATION? 


More information and links to our mailing lists, LinkedIn 
groups, and Facebook group are available at our website: 
http://www.bsdcertification.org 


Registration for upcoming exam events is available at our 
registration website: 
https://register.bsdcertification.org//register/get-a-bsdcg-id 


Linux greybeards release beta 


of systemd-free Debian fork 





\ tas debut in the 
[a name of ‘Init free- 
dom' 


| Devuan ‘Jessie’ be- 


The effort to create a systemd-free Debian 
fork has borne fruit, with a beta of “Devuan 
Jessie” appearing in the wild. 


Devuan came into being after a rebellion by a 
self-described “Veteran Unix Admin collec- 
tive’ argued that Debian had betrayed its 
roots and was becoming too desktop- 
oriented. The item to which they objected 
most vigorously was the inclusion of the sys- 
temd process manager. The rebels therefore 
decided to fork Debian and “preserve Init free- 
dom”. The group renamed itself and its distri- 
bution “Devuan” and got to work, promising a 
fork that looked, felt, and quacked like Debian 
in all regards other than imposing systemd as 
the default Init option. 


The group initially promised to deliver in 
Spring 2015. Alphas circulated during 2015, 
and in recent days betas have appeared 
here. Versions for the Raspberry Pi, Banana 
Pi and AMD64 are on offer. 


Kudos, though, to the group for getting it out 
there! Now to see if there is really a ground- 
swell of support for the cause of “Init free- 
dom”, as the greybeards name their cause. 


http://www.theregister.co.uk/2016/04/29/syste 
md_free_debian_fork_devuan_reaches_beta/ 
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To ensure an improvement per- 
formance experience for its 3.5 mil- 
lion users, Node.js Foundation has 
released Node.js Version 6 with 
Long Term Support. This release 


supports 93 percent of the ECMAS- 
a S cript 6 standard and uses Google’s 
V8 version 5.0 for the JS engine. 


Node.js, the JavaScript runtime, 
has hit version 6. The new Node.js 
v6 brings along lots of perform- 
ance and security enhancements. 





Node.js v6 is released as a long term support (LTS) release with an aim to provide a high-quality 
solution to the developers without compromising on the consistency front. Thus, Node.js 6.x will 
continue receiving official support until April 2018, and then grab only maintenance updates until 
April 2019. 


This Node.js 6 LTS release has also marked the end of Long Term Support for the Node.js 0.12 
branch. As a result, currently, the officially supported versions are 4.x and 6.x. 


Note that the Long Term Support for Node.js 4.x will end in April 2017. So, you have plenty of 
time to make the move to version 6.x LTS. 


However, Node.js 6.x will continue to remain the recommended Node.js version for the produc- 
tion usage because even though Node.js 6.x is a stable release, it will also contain new JS fea- 
tures that are under testing. 


Node.js also supports 93% of the ECMAScript 6 standard that was released in June 2015, which 
is much higher as compared to the 56% support provided by Node.js 5.x. 


The other major changes in Node.js 6.x include the usage of Google’s V8 version 5.0 for the JS 
engine and other improvements. 
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Michael W Lucas, critically acclaimed author of many BSD books, intro- 
duces the long awaited FreeBSDMichael W Lucas, critically acclaimed 
author of many BSD books, introduces the long awaited FreeBSD Mas- 
tery: Advanced ZFS co-written by Allan Jude. This book is the follow- 
up to FreeBSD Mastery: ZFS, which released last year. 


ZFS improves everything about systems administration. Once you 
peek under the hood, though, ZFS’ bewildering array of knobs and tun- 
ables can overwhelm anyone. ZFS experts can make their servers 
zing - and now you can too, with FreeBSD Mastery: Advanced ZFS. 


This small book teaches you to: 


¢ Use boot environments to make the riskiest sysadmin tasks boring 





* Delegate filesystem privileges to users 
¢ Containerize ZFS datasets with jails 
* Quickly and efficiently replicate data between machines 
¢ Split layers off of mirrors 
¢ Optimize ZFS block storage 
¢ Handle large storage arrays 
¢ Select caching strategies to improve performance 
¢ Manage next-generation storage hardware 
¢ Identify and remove bottlenecks 
¢ Build screaming fast database storage 
* Dive deep into pools, metaslabs, and more! 


Whether you manage a single small server or international datacenters, simplify your storage 
with FreeBSD Mastery: Advanced ZFS. 
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Supercharge your Raspberry Pi 3 right away 


A separate Chromium OS update has been 
made for Raspberry Pi 3 owners and for those 
who wanted to run the operating system on the 
latest iteration of Raspberry Pi 3; however, they 
need to know the following things before they 
can go ahead and download the Chromium OS 
for Raspberry Pi 3 0.5 binary image. 





Here is what is going to be included in Chromium OS for Raspberry Pi 3. 


Chromium OS for Raspberry Pi 3 includes several improvements and features. One of them hap- 
pens to be the Linux 4.2.8-ckt8 kernel with a reduced size, as well as BFS tweaks for improved 
latency, and less debugging output. Moreover, it also brings multiple improvements to the sound 
driver, as well as better storage performance; this is due to the new BFQ hierarchical scheduler 
and on demand governor tweaks. 


However, performance is not the only thing that is going to be rated here, since a fluid and clutter 
free user experience is also required in order to ensure that this update is recommended to other 
Raspberry Pi 3 owners. As of the coming of this experience, it has improved significantly, and of- 
fers fixes for the Kiosk Mode. Additionally, it also brings fixes to the VC4 GPU driver and various 
video modes, especially for those running Chromium OS for Raspberry Pi 3 on non-1080p dis- 
plays. 


Raspberry Pi 3 Model B was released on February 29, 2016, and featured a built-in Wi-Fi and Blu- 
etooth chip. Not only this, but the logic board also sported a 64-bit quad-core ARM Cortex-A53 
processing core, which would eventually deliver significant performance as compared to the logic 
board’s predecessors. While there are a ton of improvements introduced, there are also more 
than a few limitations that you should be informed about. 


First of all, the on-board Wi-Fi of the Raspberry Pi 3 Model B computer is not yet supported, and 
Netflix videos cannot be streamed as well. Another drawback is that HTML5 playback works only 
on websites that also offer Flash as an alternative to HTML5 video. If you are absolutely comfort- 
able with these limitations, then you can proceed to download the binary image right now. 
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Larry the BSD Guy, also known as the Free Software Guy, is hanging up his jersey to 
pursue another of his own writing interests. Read his final column from Fossforce, this 
one featuring LinuxFest NorthWest, an annual open-source trade show held in Belling- 
ham, Washington. Thank you for your contributions to the free and open source com- 
munity, and we wish you the best in future endeavors. 





It is a sad day at the FOSS Force office. Larry Cafiero says goodbye and walks off into the sun- 
set. 


This weekend, the Grand Old Man (or Woman — take your pick) of Linux expos in North America 
takes place in the upper left corner of the United States. 


For over a decade and a half, LinuxFest Northwest has flown the flag literally in Microsoft's back- 
yard, an annual open source event held the last weekend in April in Bellingham, Wash. LFNW fea- 
tures presentations and exhibits on various free and open source topics, as well as Linux distribu- 
tions and applications. It usually has something for everyone from the novice to the professional. 


It has a special place in my heart as well. While | think that SCALE is the best show on the conti- 
nent for obvious reasons (the SCALE Publicity Team is solely responsible, he says in jest), LFNW 
is my favorite show to attend, not only because of the history but because of the community vibe 
the show gives off at an expo that has refused to give in to the creeping corporatism to which 
other shows have succumbed .... 


The policy both reaffirms and broadens a goal laid out in the Administra- 
tion’s Second Open Government National Action Plan for "improved ac- 
cess to custom software code developed for the Federal Government." 
The Plan emphasized use of (and contributing back to) open source soft- 
ware to fuel innovation, lower costs, and benefit the public. It also furthers 
a long-standing ‘default to open’ objective going back to the early days of the Administration. 





The draft policy features several components. First, it provides a recommended "3-step process" 
on software procurement considerations to minimize procurement of custom-developed software. 
Second, it establishes requirements for releasing code in the public domain or as open source 
software, replicating early efforts by agencies to have in place policies to release code developed 


in-house. 3 
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Finally, covered agencies must deliver for reuse custom code produced in the performance of a 
federal government agreement and, subject to certain exceptions, make it broadly available 
government-wide. As widely reported, each covered agency will participate in a pilot program to 
"release at least 20 percent of its newly-developed custom code each year as open source," us- 
ing existing Open Source Initiative licenses, but leaves room for additional licenses as necessary. 


https://opensource.com/government/1 6/4/draft-policy-federal-sourcing 


GhostBSD 10.3 Enoch ALPHA1 now available 





UDF, as well as the base system being FreeBSD 10.3. 


The developers of GhostBSD have made available the first 
Io 7 35 10.3 ALPHA. This GhostBSD distribution is code-named 
Si “Enoch”. Notable changes are added support for ZFS and 


Yes we skipped 10.2 for 10.3 since FreeBSD 10.3 was coming, we thought we should wait for 
10.3. This is the first ALPHA development release for testing and debugging for GhostBSD 10.3, 
only as MATE been released yet which is available on SourceForge and for the amd64 and i386 
architectures. 


What’s new 
¢ GhostBSD now support ZFS and UFS. 
¢ The installer support encryption for ZFS 


¢ GhostBSD Software will be updated Quarterly which will bring more stability to 
GhostBSD still user will be able to change it to latest to have the latest software up 
date. 


What changed 

¢ The installer did have a big refacing plus a new slide 

¢ There is been some important fix to Networkmgr 

¢ There is been some speed improvement with Networkmgr 

¢ Some UI improvement and speed improvement with Update Station 
¢ Mate 1.12 


e New Grub theme 
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What has been fixed 


¢ Language wish GhostBSD is installed should now be fix 
¢ Wifi Problem as been fix with the update of Networkmgr 
Official announcement: http:/Awww.ghostbsd.org/10.3_alpha1 


https:/www.freebsdnews.com/2016/05/06/ghostbsd-10-3-to-add-zfs-and-udf-support-will-be-base 
d-on-freebsd-10-3/ 


PC-BSD’s Lumina Desktop 0.9.0 Environment now available 





Ken Moore, creator of the Lumina Desktop Environ- 
ment, has made available version 0.9.0. This version 
adds a new compositing effect, and more improvements 
are lined up for version 1.0. Lumina is also part of the 
PC-BSD project. 


“First, | would like to thank everyone for their patience 
as we continue working toward the first non-beta re- 
lease of the Lumina desktop. We are still planning on 
version 1.0.0 getting released later this year (aligning 
with the FreeBSD 11.0 Release Schedule or earlier), but 
some issues have come to light that required we adjust 





our feature list for version 1.0.0 a bit.” 


Official announcement: 
http:/umina-desktop.org/new-release-schedule-and-lumina-desktop-0-9-0-released/ 


https:/www.freebsdnews.com/2016/05/06/pc-bsds-lumina-desktop-0-9-0-environment-launches- 
with-compositing-effects/ 
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The most significant changes in this release is a rewrite of the webGUI utilizing Bootstrap, and 
the underlying system, including the base system and kernel, being converted entirely to 
FreeBSD pkg. The pkg conversion enables us to update pieces of the system individually going 
forward, rather than the monolithic updates of the past. The webGUI rewrite brings a new respon- 
sive look and feel to pfSense requiring a minimum of resizing or scrolling on a wide range of de- 
vices from desktops to mobile phones. 


For the highlights, check out the Features and Highlights video. Past blog posts have covered 
some of the changes, such as the performance improvements from tryforward, and the webGUI 
update. 


To get to a release, we have closed 760 total tickets. While the majority of these were related to 
the Bootstrap conversion, 137 are fixed bugs impacting 2.2.6 and earlier releases. 


Upgrade Considerations 


As always, you can upgrade from any prior version directly to 2.3. The Upgrade Guide covers eve- 
rything you well need to know for upgrading in general. There are a few areas where additional 
caution should be exercised with this upgrade. 


Known Regressions 


¢ OpenVPN topology change — configuration upgrade code was intended to set upgraded 
OpenVPN servers to topology net30, rather than the new default of topology subnet. This is not 
working as intended in some cases, but has been fixed for 2.3.1. In the meantime, editing your 
OpenVPN server instance and setting the topology to “net30”, will accomplish the same thing 
and fix it. 


¢ IP aliases with CARP IP parent lose their parent interface association post-upgrade. Go to Fire- 
wall>Virtual IPs, edit the affected IP alias, pick the appropriate CARP IP parent, then save and 
apply changes. Make sure every virtual IP has something shown in the Interface column on 
firewall_virtual_ip.php. 


¢« Psec IPComp does not work. This is disabled by default. Disable IPComp under VPN>IPsec, 
Advanced to work around if you have enabled |[PComp. Bug 6167 


¢ IGMP Proxy does not work with VLAN interfaces. Bug 6099. This is a little-used component. If 
you are not sure what it is, you are not using it. 
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Clear Browser Cache 


Due to the many changes in the web interface, clearing your browser cache or doing a forced re- 
load (shifttrefresh) is a good idea after upgrading. If you see any cosmetic problems in the web 
interface post-upgrade, a stale browser cache is the likely reason. 


Packages 


The list of available packages in pfSense 2.3 has been significantly trimmed. We have removed 
packages that have been deprecated upstream, no longer have an active maintainer, or were 
never stable. A few have yet to be converted for Bootstrap and may return if converted. pfSense 
software is Open Source 


For those who wish to review the source code in full detail, the changes are all publicly available 
in three repositories on Github. 2.3-RELEASE and is built from the RELENG 2 3 _0 branch of 
each repository. 


Supporting the Project 


Our efforts are made possible by the support of our customers and the community. You can sup- 
port our efforts via one or more of the following ways: 


¢ pfSense Store — official hardware, apparel, and pre-loaded USB sticks direct from the source. 
Our pre-installed appliances are the fast, easy way to get up and running with a fully-optimized 
system. All are now shipping with 2.3 release installed. 


¢ Gold subscription — Immediate access to past hang out recordings , as well as the latest version 
of the book after logging in to the members area. 


¢ Commercial Support — Purchasing support from us provides you with direct access to the 
pfSense team. 


¢ Professional Services — For more involved and complex projects outside the scope of support, 


our most senior engineers are available under professional services. 


https://blog.pfsense.org/?p=2008 
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Adding ZFS to the FreeBSD Dual- 
controller Storage Concept 


by Mikhail E. Zakharov 


We should name our small project. Something in honor of 
Beastie the daemon and the large BSD operating system 
family should be reasonable, so let’s call our storage project 
the BeaST or even shorter, the BST. 


The environment for our testing purposes will be similar to 
previous ones. It is still my laptop which runs Oracle VM Vir- 
tualBox and a USB memory-sstick to store shareable virtual 
drives and slow down their IO. 


We will need to create three virtual machines. One of them (clnt-1) will be the client for our stor- 
age system. We can easily take its configuration as-is from the previous test environment. 


The last two machines (ctrl-a and ctrl-b) will serve as the storage controllers. These machines 
must be configured with at least 2,048 MB of memory to run all our tests with ZFS without issues. 


With the help of the VirtualBox Virtual Media Manager, we should configure and attach to both 
storage controllers four fixed-sized shareable drives (d00, d01, d10, d11) for ZFS data volumes 
and four fixed-sized shareable drives (f00, f01, f10, f11) which we will use for ZFS cache. 


We do not have any real hardware for our tests, so let’s pretend that the data-drives are taken 
from the shared SATA shelf and imagine our cache-drives are fast Solid State Drives (SSDs) in- 
serted into the SAS enclosure. Therefore, do not forget to add appropriate controllers to the 
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virtual-hosts configurations. Bear in mind, while the data-drives can be any size you prefer, the 
cache-drives must be at least 64 MB each. 


Although it is not really mandatory for our purposes, you can even check the “Solid State Drive” 
option for the cache-drives to assure yourself everything is done the best way possible. 























B General Storage 
System 
Display Storage Tree 
a) Beene & Controller: IDE 
Audio @) v5.2-ctrl-1-disk1.vdi 
=F Network © Empty 
6 Serial Ports & Controller: SATA 
@ uss @ doo.vdi 
Shared Folders @ do1.vdi 

@ a10.vdi 

B® a1.vai 

& Controller: SAS 

® tor.vai 

@ f1o.vdi 

® fivdi 

Ee a@ ec 








(o| Help | 


Attributes 


Hard Disk: |SAS Porto  <¢| @v 
() Solid-state Drive 


When checked the 
¢ guest system will see 
the virtual disk as a 
solid state device. 
02.00 MB 










Information 
Type (Format): 
Virtual Size: 
Actual Size: 1 
Details: Fixed size storage 
Location: /media/BeaST/f00.vdi 
Attached to: v5.2-ctrl-1, v5.2-ctrl-2 
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See Figure 1 with the screenshot of Oracle VM 
VirtualBox Manager GUI showing the storage 
configuration section of the ctrl-1 machine. 


Figure 1. Controller storage layout example. 


The network configuration is not changed. We 


will use two LAN connections: “private” for inter-controller and “public” for host-to-controllers com- 


munications. 


Latest FreeBSD 10.3 Release can be installed on the dynamic-sized drives of all three virtual ma- 


chines. 


Configuration summary is shown in the table below: 







Inter-controller 
(private) network. 
Host-only adapter 
(vboxnet0) 






IP: 192.168.56.10 


Mask: 255.255.255.0 


IP: 192.168.56.11 


Mask: 255.255.255.0 









Public network. Host- 
only adapter 
(vboxnett ) 








IP: 192.168.55.10 


Mask: 255.255.255.0 


IP: 192.168.55.11 


Mask: 255.255.255.0 









IP: 192.168.55.20 


Mask: 255.255.255.0 
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Base memory 


Shareable, fixed-sized 
virtual drives for ZFS 
data volumes on the 
SATA controller. 


Shareable, fixed-sized 
virtual drives for ZFS 
cache on SAS 
controller 


2048 MB or more 


d0O, dO1, d10, d11 — 
each drive is 100 MB 
size or more 


f00, f01, f10, f11 — at 
least 64 MB each 


2048 MB or more 


doo, d01, d10, d11 — 
each drive is 100 MB 
or more 


f00, f01, f10, f11 — at 
least 64 MB each 


Any appropriate value 
starting with 512 MB 
will do 





System virtual drives 
(Dynamic-sized) on 
the IDE controller 





At least 5 GB to store 
FreeBSD 10.3- 
Release default 
installation 


At least 5 GB to store 
FreeBSD 10.3- 
Release default 
installation 








At least 5 GB to store 
FreeBSD 10.3- 
Release default 
installation 





Install FreeBSD on the ctrl-a and the ctrl-b virtual machines using default parameters and adaO 
(dynamic-sized drive on the IDE controller) as the drive for the root file system. Then configure 
general parameters in /etc/rc.conf. This file for each controller can be easily taken from the previ- 


ous environment: 
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hostname="ctrl-a" hostname="ctrl-b" 


ifconfig_ em0="inet 192.168.56.10 netmask ifconfig_em0="inet 192.168.56.11 netmask 
255.255.255.0" # Inter-controller LAN 255.255.255.0" # Inter-controller LAN 


ifconfig _em1="inet 192.168.55.10 netmask ifconfig _em1="inet 192.168.55.11 netmask 
255.255.255.0" # Public network 255.255.255.0" # Public network 


sshd_enable="YES" sshd_enable="YES" 


# Set dumpdev to "AUTO" to enable crash # Set dumpdev to "AUTO" to enable crash 
dumps, "NO" to disable dumps, "NO" to disable 


dumpdev="AUTO" dumpdev="AUTO" 
# VirtualBox guest additions # VirtualBox guest additions 
vboxguest_enable="YES" vboxguest_enable="YES 


vboxservice_enable="YES" vboxservice_enable="YES" 


# iSCSI # iSCSI 


ctld_enable="YES" # Targets ctld_enable="YES" # target 








iscsid_enable="YES" # Initiators iscsid_enable="YES" # initiator 


Do not forget to set iSCSI “disconnection on fail’ kernel variable in /etc/sysctl.conf on both sys- 
tems to be able to failover in case of disaster to the alive controller: 


[eS ra Ueno ersalee eelal Level ql lo uleslerol quae cael qi 
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After reboot check if everything runs well. Note that shared data-drives should be recognized by 
the kernel as ada1, ada2, ada3, ada4 and shared cache-drives would be daO, da1, da2, da3: 


root@ctrl-a:/home/beast # dmesg | grep "da[0-9]:" 


dail: }<VBOxX HARDDISK 1.0> Fixed Direceh Access SEC-3 SCol device 
da0: 300.000MB/s transfers 

da0O: Command Queueing enabled 

daO: 1LOOMB (204800 512 byte sectors) 

dal? <VBOX HARDDISK L.0> Faxed Direct Access SPC-3  SsCol device 
dal: 300.000MB/s transfers 

dal: Command Queueing enabled 

dal: 1LOOMB (204800 512 byte sectors) 

Olea ee Als (Op.Cs VN 2 IDB Med .Ce lle 012 200) Sisle-(=\o ui Balralsveusy Welelovsnca aC one Oo 1b 

da2: 300.000MB/s transfers 

da2: Command Queueing enabled 

da2: 1LOOMB (204800 512 byte sectors) 

lel eae GVAslOp Cosh Ne UDB IG Si. wile OP: aal daly (=) lei sb ia'= (ees wAeles—sstonivs = Ca oee Oe lionel =piaiker= 
da3: 300.000MB/s transfers 

da3: Command Queueing enabled 

da3: 1LOOMB (204800 512 byte sectors) 

adaOQ: <VBOX HARDDISK 1.0> ATA-6 device 

adaOQ: Serial Number VB9c2e46e9-3f3664e0 

adaQ: 33.300MB/s transfers (UDMA2, PIO 65536bytes) 

adaO: 20480MB (41943040 512 byte sectors) 


adaOQ: Previously was known as ad0O 
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<VBOX HARDDISK 10> ATA-—6 SATA 2 7x device 

Serial Number VB9508e565-d9dfd8c7 

300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) 
Command Queueing enabled 

1LOOMB (204800 512 byte sectors) 

Previously was known as ad4 

<VBOX HARDDISK 1.0> ATA-6 SATA 2.x device 

Serial Number VB6a067/a06-5a2a2e74 

300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) 
Command Queueing enabled 

LOOMB (204800 512 byte sectors) 

Previously was known as ad6 

<VBOX HARDDISK 1.0> ATA-6 SATA 2.x device 

S\eueabeMly IN bialeysue yids 10)7 1 IRW Cats) Ce Il Sie: 

300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) 
Command Queueing enabled 

LOOMB (204800 512 byte sectors) 

Previously was known as ad8 

<VBOX HARDDISK 1.0> ATA-6 SATA 2.x device 


Serial Number VBccf97b29-all1l4/aa3 


300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) 


Command Queueing enabled 
LOOMB (204800 512 byte sectors) 


Previously was known as adl0 
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ZFS pools basic configuration 


If everything is done properly, we can configure ZFS pools on the controllers. 


Let’s use ada1/ada2 drives to create ctrl_a_m0O pool on the ctrl-a controller and ada3/ada4 drives 
to form ctrl_b m0 pool on the ctrl-b controller. Then add volumes — vO to each pool: 










zpool create -m none ctrl-a_m0 /dev/ada1 / 
dev/ada2 
zfs create -V 120M ctrl-a_m0/vO 


dev/ada4 

zfs create -V 120M ctrl-b_m0O/vO 
The only interesting options are: 

-m none — prevents occasional pool mounting. 


-V —- desired volume size. 


To check the result, run: 


root@ctrl-a:/home/beast # zpool status 
Kolo ere alleen ie) 
SiGe Se LOIN [le aay et 
scan: none requested 


GrOlgne alot 


NAME STATE READ WRITE ChSUM 


oie rein ONLINE 


NOkomn ONLINE 


Oko wA ONLINE 


zpool create -m none ctrl-b_ m0 /dev/ada3 / 
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Read and write caches 


Now let’s try to move ZFS cache to the shared devices. In our case these will be daO — da3 
drives, which we agree to consider the fast SSDs. 


First, we will try to implement quazi-write cache with ZFS Intent Log (ZIL) forcing it to handle both 
types of synchronous and asynchronous transactions and write them to the shared drive da0O (the 
ctrl-a_m0 pool on the ctrl-a controller) and da2 (ctrl-b_ m0 pool on on the ctrl-b): 


ctrl-a 


zpool add ctrl-a_m0 log /dev/da0 


# Always write and flush all file system 
transactions. 
zfs set sync=always ctrl-a_m0O 


ctrl-b 


zpool add ctrl-b_m0 log /dev/da2 


# Always write and flush all file system 
transactions. 





zfs set sync=always ctrl-b_m0O 





Second, add the shared drives to the pools: da‘ (on the ctrl-a) and da3 (on the ctrl-b) to use as 
the read-caches for both pools. Finally, disable in-memory read-cache completely and move it to 


the shared devices. 
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ARC / L2ARC Configuration (read cache): 


ctrl-a 


zpool add ctrl-a_m0O cache /dev/da‘ 


# Turn-off ARC caching 
zfs set primarycache=none ctrl-a_m0O 


# Enable L2ARC caching to the shared device 
zfs set secondarycache=all ctrl-a_m0 


ZFS pools final configurations 


ctrl-b 


zpool add ctrl-b_m0O cache /dev/da3 
# Turn-off ARC caching on ctrl-b 
zfs set primarycache=none ctrl-b_m0O 


# Enable L2ARC caching to the shared device 





zfs set secondarycache=all ctrl-b_m0O 


On the last step of ZFS configuration we should import pools from opposite controllers. Then we 


have to set “failmode=continue” property. 


This is quite dangerous because we just reject to stop working even if any errors are detected on 
the pools, but it is necessary to be able to failover to the next controller in case of disaster. So: 


ctrl-a 


Zpool import -N ctrl-b_m0O 


zpool set failmode=continue ctrl-a_m0O 
zpool set failmode=continue ctrl-b_m0O 


ctrl-b 


Zpool import -N ctrl-a_m0 


zpool set failmode=continue ctrl-a_m0O 





zpool set failmode=continue ctrl-b_m0O 


-N option of “zpool import” command above prevents the pool from being mounted. 


Finally, check zpool configuration. We should be able to see both pools on both storage control- 


lers: 
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NAME 


STATE READ WRITE CKSUM 


Geni aime ONLINE 


adal ONLINE 


ada2 ONLINE 


imelers 


da0 ONLINE 


cache 


cle ONLINE 


errors: 
pool: 
Seake: 
SKe/shlaer 
(ELON ine aLio( 
NAME 


Gee 


No known data errors 
eile 
ONLINE 


none requested 


oe awit READ WRITE 


-b m0 ONLINE 


ada3 OMNES 


ada4 ONLINE 


OCS 


da2 ONLINE 


cache 


da3 ONLINE 


Sic Ig@Oic S 5 


No known data errors 
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The Arbitrator 


The arbitrator mechanism is similar to the one that was discussed in the previous paper. The only 
difference is that now we use data-volumes on ZFS pools. So just edit /etc/ctl.conf to add appropri- 
ate inter-controller communication configurations for the vO volumes: 


ctrl-a ctrl-b 
portal-group pgO { portal-group pgO { 

discovery-auth-group no-authentication 

listen 192.168.56.10 discovery-auth-group no-authentication 
} 


listen 192.168.56.11 


target iqn.2016-01.local.sss.private:targetO { |} 
auth-group no-authentication 


portal-group pgO target iqn.2016-01.local.sss.private:targetO { 
lun 0 { auth-group no-authentication 
path /dev/zvol/ctrl-a_m0/vO 
} portal-group pgO 
lun O { 


path /dev/zvol/ctrl-b_m0/vO 








} 


Then restart ctld daemon on both controllers to update iSCSI targets: 





service ctld restart service ctld restart 


From both controllers connect with the LUNs on the opposite controllers: 


iscsictl -A -p 192.168.56.11 -t iqn. iscsictl -A -p 192.168.56.10 -t iqn. 
2016-01 .local.sss.private:targetO 2016-01 .local.sss.private:targetO 
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And finally, assemble the arbitrating construction by creating active-passive multipath devices, 
which have active paths pointing to the opposite controllers at their initial state: 


gmultipath create CTRL_B_ BACK /dev/da4/ | gmultipath create CTRL_A_BACK /dev/da4 / 
dev/zvol/ctrl-b_m0/vO dev/zvol/ctrl-a_m0/vO 





Front-end configuration 


The tandem of the arbitrator and the external shared cache forms a reliable structure resistant to 
a single controller failure. Therefore we can continue to configure host-connection configuration 
and update /etc/ctl.conf with the “public” sections to allow host access: 








portal-group pg0O { portal-group pgO { 
discovery-auth-group no-authentication 
listen 192.168.56.10 discovery-auth-group no-authentication 
} 
listen 192.168.56.11 
portal-group pg1 { } 
discovery-auth-group no-authentication 
listen 192.168.55.10 portal-group pg1 { 
} 
discovery-auth-group no-authentication 
target iqn.2016-01.local.sss.private:target0 { listen 192.168.55.11 
auth-group no-authentication 
portal-group pgO } 
lun 0 { target iqn.2016-01.local.sss.private:targetoO { 
path /dev/zvol/ctrl-a_m0/vO a 
} auth-group no-authentication 
portal-group pgO 
lun O { 
path /dev/zvol/ctrl-b_m0O/vO 
} 
} 
25 
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ctrl-b 


target iqn.2016-01.local.sss.public:targetO { target iqn.2016-01.local.sss.public:targetO { 
auth-group no-authentication 
portal-group pg1 auth-group no-authentication 


lun O { portal-group pg1 
path /dev/zvol/ctrl-a_m0O/vO 
} lun 0 { 


lun 1 { path /dev/zvol/ctrl-b_m0/vO 
path /dev/multipath/CTRL_B_ BACK ’ 
} 


lun 1 { 


path /dev/multipath/CTRL_A_BACK 








Then force ctld daemon to re-read and refresh iSCSI targets configuration on both controllers: 





killall -HUP ctld killall -HUP ctld 


Check for essential messages in the dmesg output to see if everything goes smooth: 
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GEOM MULTIPATH: CTRL _B BACK created 
GEOM MULTIPATH: da4 added to CTRL _B BACK 


GEOM MULELPATH: da4 is mow aceive: pati ineCIRL B BACK 


GEOM MULTIPATH: zvol/ctrl-b m0/v0 added to CTRL _B BACK 





Figure 2 shows our storage system architecture layout we just created: 











ctrl-a_m0 
iSCSI ! ti 
Public CTRL_B_BACK ' ' 
multipath arbitrator 


a 
a0 
oO 


N 
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ctrl-b_m0O/vO 







ISCSI Physical 
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A BACK multipath 
arbitrator.” 


~-_~ 
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=a 
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ctrl-a_m0/vO 





ISCSI Private —— Physical 


iSCSI 
Public 


ctrl-b_m0O 


Figure 2. Dual-controller storage architecture with ZFS and external cache overview. 


The client side 


As the storage controllers are ready to serve requests, we can prepare the client. The clnt-1 initial 
configuration is identical to the previous environment and can be used as-is. The essential lines 
of /etc/rc.conf are shown below: 


hostname="clnt-1" 


ieeforaarale p= ihe) Valse ML SA Peis es S95 40) Mal =Ieiiiciol ys Soy AS yo) OMe nis Nedolonlaes ag) =) 0 


work 
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Sts) slo eels Nad Se 


# Set dumpdev to "AUTO" to enable crash dumps, "NO" to disable 


dumpdev="AUTO" 
# VirtualBox guest additions 
Wleorde bio sse, (eialeie dog oy 


Cer Seieleles ISinevloliS iy Vala, oy 


ele cal 


asKersahe| Veiniel Ge cg, Sy ge dUigamieakchele ars, 
The /etc/sysctl.conf file is also the same: 


Stew @iptlesterstall Scceil iy vero sie aleseeigueleeueaketal— 


After the basic client preparation we can try to connect with public iSCSI targets of both storage 
controllers: 


Tees als( 4 Bl hue I we aie) alse aalicyomyectanegd Mie 10, alo igi ON cc at GM rp berercmiLatcvcucra: onvlloiesl(e; eer lao fo1e,0) 


dirstGhs wig Gs Sky tay! \e om alec a etcneo ey aN seek oiginy, ONal ONL a oreeamictsrs Sebledise seca (= a0) 


Check if dmesg output shows the appearance of the new daO, da1, da2, and da3 drives: 


igor elCreIMie= i) ey aveiileylol-velsie © 7; aelileicen yi chacien Mo voeu 2) G2 


dali: <FREEBSD CTLDISK O001> Fixed Direct Access SPC-—4 SCSI device 





"a 


Serial Number MYSERIAL 

150.000MB/s transfers 

Command Queueing enabled 

OAC Ist CZ NS ANON OMA Ioniel= mei eneenacrs sor Nal GS Al Ee) 

<FREEBSD CTLDISK QOO01> Fixed Direct Access SPC-4 SCSI device 
Serial Number MYSERIAL Z 

150.000MB/s transfers 


Command Queueing enabled 





P2OMBY (2457605512 byte sectors: 64H) 325/71 120C) 

<FREEBSD CTLDISK O001> Fixed Direct Access SPC-4 SCSI device 
Serial Number MYSERIAL 

150.000MB/s transfers 

Command Queueing enabled 

IR ZAO Tei CEs I AolO Monae Wm axci= Gmeiac Heller Nai s. 37 AU el Zee.) 
<FREEBSDSCTEDISk QO001> Fixed Direce Access SPC—4  SCal device 
Serial Number MYSERIAL Zs 

150.000MB/s transfers 

Command Queueing enabled 


IPA OSI 2 eo Ol tes IA einer eta 16) clanact ae Neb uey ac le lLyA0G)) 


Then create appropriate multipathing devices to access both storage controllers: 


gmultipath create CTRL A /dev/da0 /dev/da3 


gmultipath create CTRL B /dev/da2 /dev/dal 
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Finally we can create a striped volume, then add and mount a filesystem: 


If everything is well, we will see the appearance of the new mounted filesystem: 


Then we can run all the tests we got familiar with in the previous version of the BeaST system. 
So let’s start with a file copy operation: 


Figures 3, 4, and 5 show us the state of both controllers and the client: 


File Edit View Search Terminal Tabs Help 








ctri-a * | ctrl-b x | clnt-1 x 
capacity operations bandwidth 
pool alloc free read write read write 
ctrl-a_m0o 646K 159M 0 3 3.30K 328K 
adai 288K 79.7M 0 1 102 153K 
ada2 359K 79.6M 0 0 3.20K 31.5K 
logs - - - - - - 
da@ 384K 79.6M 0 1 @ 143K 
cache - - - - - - 
dai 32K 95.5M 0 0 0 0 
ctr1-b_m0d 306K 160M 0 0 0 0 
ada3 110K 79.9M 0 0 0 0 
ada4 196K 79.8M 0 0 0 0 
logs - - - - - - 
da2 0 80M 0 0 0 0 
cache - - - - - - 
da3 18.5K 95.5M 0 0 0 0 
i WY) 
Figure 3. The ctrl-a normal operations. 
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You can see “zpool iostat -v 5” on Figure 3, which shows us the activity on ctril-a_m0 pool de- 
tected by the ctrl-a. At the same time workload of the ctrl-b_m0 pool is hidden to the ctrl-a control- 
ler. Actually both pools are utilized but from different controllers. 


File Edit View Search Terminal Tabs Help 


The same situation on the 








ctri-a 3¢ | ctri-b * | clnt-1 





capacity operations bandwidth 
pool alloc free read write read write 
ctrl-a_m0 284K 160M 0 0 0 0 
ada 108K 79.9M 0 0 0 0 
ada2 176K 79.8M 0 0 0 0 
logs - - - - - - 
dad 0 80M 0 0 0 0 
cache - - - - - - 
dai 18K 95.5M 0 0 0 0 
ctrl-b_m@ 4.59M 155M 0 2 @ 101K 
ada3 2.30M 77.7M 0 0 0 610 
ada4 2.29M 77.7M 0 0 0 406 
logs - - - - - - 
da2 6.254 73.8M 0 1 Q@ 100K 
cache - - - - - - 
da3 32K 95.5M 0 0 0 305 
i 


Figure 4. The ctrl-b normal operations. 


From the client side (Figure 5) both primary 
both owner controllers. 


File Edit View Search Terminal Tabs Help 


ctrl_b controller (Figure 4): 
the ctrl-b_m0O pool activity is 
shown, but the ctrl-a_m0O 
workload is not detected. 








paths are active and the data goes normally though 


Now let’s actually fail one of 





the controllers and see the 








ctrl-a x ctri-b x cint-1 

cdo 0.0 0.0 0.0 0.0 0 0.0 0 
pass 0.0 0.0 0.0 0.0 0 0.0 0 
passi 0.0 0.0 0.0 0.0 0 0.0 0 
pass2 0.0 0.0 0.0 0.0 0 0.0 0 
pass3 0.0 06.0 0.0 0.0 0 0.0 @ 
pass4 0.0 0.0 0.0 0.0 0 0.0 0 
pass5S 0.0 0.0 0.0 0.0 0 0.0 0 

extended device statistics 

device r/s w/s kr/s kw/s qlen svc_t %b 
ada@ 0.0 0.0 0.0 0.0 0 : 0 
da 0.0 0.2 0.0 12.8 9 1611.1 32 
dai 0.0 0.0 0.0 0.0 0 0.0 0 
da2 0.0 a.8 0.0 134.2 4 3153.0 124 
da3 0.0 0.0 0.0 0.0 0 0.0 0 
cdo 0.0 0.0 0.0 0.0 0 0.0 0 
passO@ 0.0 0.0 0.0 0.0 0 0.0 0 
passi 0.0 0.0 0.60 0.0 0 0.0 0 
pass2 0.0 90.0 0.0 0.0 0 0.0 @ 
pass3 0.0 0.0 0.0 0.0 0 0.0 0 
pass4 0.0 0.0 0.0 0.0 0 0.0 0 
pass5S 0.0 0.0 0.0 0.0 6 0.0 20 


Figure 4. The ctrl-b normal operations. 


result. Traditionally | will 
kick-off the ctrl-a. 
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Figure 6 now shows the survived controller: 


File Edit View Search Terminal Tabs Help 














x 
capacity operations bandwidth 
pool alloc free read write read write 
ctrl-a_m0 478K 160M 0 3 0 303K 
adai 133K 79.9M 0 0 Q@ 42.8K 
ada2 345K 79.7M 0 1 @ 152K 
logs - - - - - - 
da@ 260K 79.7M 0 1 Q@ 108K 
cache - - - - - - 
dai 17.5K 95.5M 0 0 0 0 
ctrl-b_mO 30.0M 130M 1 2 4.76K 89.0K 
ada3 14.54 65.5M 0 0 3.47K 1.69K 
ada4 15.5M 64.5M 0 Q@ 1.29K 812 
logs - - - - - - 
da2 2M 78M 0 0 Q@ 86.5K 
cache - - - - - - 
da3 37K 95.5M 0 0 0 304 
fi 


Figure 6. The ctrl-b after ctrl-a failed. 


Now the ctrl-b takes all the clients workload: the arbitrator directed all data traffic through the sur- 
vived controller. 








File Edit View Search Terminal Tabs Help And it is not really a sur- 
prise as the clnt-1 has 
aaa or aa Te aS lost all paths to the ctrl-a. 
extended device statistics On Figure 7 you can see 
device r/s w/s kr/s kw/s qlen svc_t %b 
ada@ 8.0 06.0 8.0 0.0 6 66 @ that daO and dai have 
da2 8.0 0.6 @.0 19.1 16 10460.7 163 
da3 0.0 0.2 0.0 0.8 6 7662.1 86 disappeared while da2/ 
cdo 8.6 06.0 8.0 960 GO 66 6 da3 is taking all the work- 
pass@ 8.0 0.0 8.0 0.0 6 66 @ 
pass 0.0 0.0 0.0 0.0 O60 6.0 @B load. 
pass4 8.0 0.0 8.0 0.0 6 66 @ 
pass5 8.0 0.0 8.0 0.0 6 66 @ 
extended device statistics 
device r/s_ w/s kr/s kw/s glen svc_t %b 
ada@ 8.0 0.0 8.0 8.0 0 @ 
da2 8.0 3.0 8.0 95.6 19 12404.8 79 
da3 8.0 0.2 @.0 12.8  841715.8 34 
cdo 8.0 0.0 8.0 0.0 6 66 @ 
pass 8.0 0.0 8.0 8.0 6 606 B 
pass 8.0 0.0 8.0 0.0 6 66 @ 
pass4 8.0 0.0 8.0 00 6 66 @ 
pass5 8.0 0.0 8.0 0.0 6 660 @ 





Figure 7. The client cint-1 after ctrl-a failure. 
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After the file-copy operation has been finished, we can check if the data are written correctly: 


rook@clint—-ls/home/ beast # mds ports tgz 
UIDs e Geroneie susie py Pe a owarcholelleicii| tele ctors)leN/cloglicio cue 4 em clei CfeKel oye) 
root @clnt—-l:/home/beast # mds /Storage/pores. tgz 


MD5 (/storage/ports.tgz) = 82a5d6a7a3a89b7a5185a543fab6b3a56 


We can state that the system works well in the laboratory. But now we are on a very slippery 

floor, as ZFS is not a cluster system and therefore is not designed to be run in the shared environ- 
ments. Our storage solution is still for testing purposes only. So nobody can guarantee that every- 
thing will work in production. Beware of data loss! 


The recovery procedure 


From any point of view this is definitely not the brightest side of life. Actually, it is painful because 
the most important thing we should remember now, is that we will lose data if the last survived 
controller drops its access to the pools. In other words, the last controller must stay online until 
the recovery is finished: no shutdowns at all, even planned one! 


Let’s study the case. First of all, we must check the pool status: 


root@ctrl-b:/home/beast # zpool status 
poe: -crrl—a ml 
Stace “ONLINE 
scan: none requested 
Cont UG: 
NAME oy iy wah READ WRITE CKSUM 
Gwalior ine ONLINE 


adal ONLINE 
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Neto, ONLINE 
rogs 

erene) ONLINE 
cache 

orome ONLINE 


errors: No known data errors 


Sore Ee ener aiken ong ie) 


Stace: ONLINE 
scan: none requested 


(Soins 


STATE READ WRITE CKSUM 


ONLINE 


ONLINE 


ONLINE 


ONLINE 


ONLINE 


errors: No known data errors 
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But we must not trust it at all! As we remember, one controller cannot see pool activities of the 
other controller. Additionally, we disabled error detection on the pools by the “failmode=continue” 
option with our own hands. 


So let’s run the “scrub” procedure on the pool attached to the failed controller, then check the 
status to see the real picture of disaster: 


root@ctrl—b: /home/beast # zpool scrub ctri-a m0 
ReOtGctri—w-) home/ beast 4 7 >Ool Status, —v, 

[Seren es aene alka. me ie) 

Steere ONT NE 


status: One or more devices has experienced an error resulting in 
okchere! 


corruption. Applications may be affected. 


action: Restore the file in guestion if possible. Otherwise restore 
ia ake 


entire pool from backup. 
Sialoie 


Scan: serub repaired 1K in OhOm with 2 errors on Tue Apr 19 
hoy 239 32 0a 


Contig: 


NAME STATE READ WRITE CKSUM 


Guedes ne ONLINE Zz 


adal ONLINE 

NO reW, ONLINE 
logs 

da0O ONLINE 


cache 





7a 


ONLINE 


errors: Permanent errors have been detected in the following files: 
<metadata>:<Oxlb> 


<metadata>:<0x20> 


pool: Ferri bom’ 
SieelBlows LO) BIEN ch 
SSEUG  aloigls® sas ejb is) cS: 


(Scone meakre(” 


STATE READ WRITE CKSUM 
ONLINE 
ONLINE 


ONLINE 
ONLINE 
ONLINE 


errors: No known data: errors 


Finally, run scrub on the ctrl-b_ m0 pool and try to boot the ctrl-a controller. 


Immediately after the ctrl-a is booted, login to its console and run “zpool export” to disconnect the 


controller from both of the pools: 
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root@ctrl—-—a:/home/beast # zpool export ctrl-a m0 


root@ctrl-a:/home/beast # zpool export ctril-b m0 





Now we can breath normally. Calmly reboot crtl-a once more, then import pools back again: 


root@ctrl-a:/home/beast # ua elona so \pmeheiadl ral 10 


root@ctrl-a:/home/beast # rmpowe, Nice rl— m0 





Then run scrub on both controllers: 


root@ctrl-a:/home/beast # Ctril-at im 
root@ctrl-a:/home/beast # el cagih omuine 
root@ctrl-b:/home/beast # cecil e ane 


root@ctrl—-b:/home/beast # Simaad enya 6 





Also you may need to clear errors on the pools: 


VA OO Only Gl erolig wolenelk fei) 51010) 


ZDOOUGleat eer ly lam 





If we succeed with the pools recovery procedure we should carefully restore the arbitrator, all bro- 
ken iSCSI connections, and multipathing primary paths. In other words we have to repeat most of 
the steps specified in the article once more. It is a very tedious process, but it should be done 
carefully to finish the storage system restoration. 


37 


BSD 


MAGAZINE 





What is next 


We have added ZFS to the BeaST project and that is very good. But bearing in mind the real life 
and performance demands of the different workloads, it is probably not the best idea to put the 
main system cache to the external solid state drives. Better if we are able to implement in- 
memory cache and make it somehow to be mirrored between both controllers. This would be the 
first task for the future development. 


Second, we still must test the BeaST project under the pressure of high workloads on the real 
hardware and test not only the performance but the stability of our storage concept. 


Third, FreeBSD 10.3 Release brought to the table the high-availability options for the CAM Target 
Layer and we should check how the BeaST can benefit from the new features. 


About the Author: 
| am a proud SAN/storage IBMer. 10 years of experience | 
in large SAN and storage environments: mainly Hitachi, 


HP and Brocade. Empty — expect-like tool author. 
FreeBSD enthusiast. 


Mikhail E. Zakharov, zmey20000@yahoo.com 
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HOW IMPORTANT IS YOUR DATA? 


Years of family photos. Your entire music 
and movie collection. Office documents 
you've put hours of work into. Backups for 
every computer you own. We ask again, how 
important is your data? 


NOW IMAGINE LOSING IT ALL 


Losing one bit - that’s all it takes. One single bit, and 
your file is gone. 





The worst part? You won't know until you 
absolutely need that file again. Example of one-bit corruption 


THE SOLUTION 


The FreeNAS Mini has emerged as the clear choice to The Mini boasts these state-of-the- 
save your digital life. No other NAS in its class offers art features: 
ECC (error correcting code) memory and ZFS bitrot 


protection to ensure data always reaches disk * 8-core 2.4GHZ Intel® Atom™ processor 
: : . + Up to 16TB of storage capacity 
without corruption and never degrades over time. 


+ 16GB of ECC memory (with the option to upgrade 
to 32GB) 


No other NAS combines the inherent data integrity + 2x 1 Gigabit network controllers 
+ Remote management port (IPM) 


- Tool-less design; hot swappable drive trays 
encryption. No other NAS provides comparable power - FreeNAS installed and configured 


and flexibility. The FreeNAS Mini is, hands-down, the 
best home and small office storage appliance you can 
buy on the market. When it comes to saving your 
important data, there simply is no other solution. 


and security of the ZFS filesystem with fast on-disk 





systems 


aad 


CERTIFIED 
STORAGE 


With over six million downloads, 
FreeNAS is undisputedly the most 
popular storage operating system 
in the world. 


Sure, you could build your own FreeNAS system: 
research every hardware option, order all the 

parts, wait for everything to ship and arrive, vent at 
customer service because it hasnt, and finally build it 
yourself while hoping everything fits - only to install 
the software and discover that the system you spent 
days agonizing over isn’t even compatible. Or... 


MAKE IT EASY ON YOURSELF 


As the sponsors and lead developers of the FreeNAS 
project, iXsystems has combined over 20 years of 
hardware experience with our FreeNAS expertise to 
bring you FreeNAS Certified Storage. We make it 
easy to enjoy all the benefits of FreeNAS without 
the headache of building, setting up, configuring, 
and supporting it yourself. As one of the leaders in 
the storage industry, you know that you're getting the 
best combination of hardware designed for optimal 
performance with FreeNAS. 


Every FreeNAS server we ship is... 


» Custom built and optimized for your use case 

» Installed, configured, tested, and guaranteed to work out 
of the box 

» Supported by the Silicon Valley team that designed and 
built it 

» Backed by a 3 years parts and labor limited warranty 


http://www.iXsystems.com/storage/freenas-certified-storage/ 





As one of the leaders in the storage industry, you 
know that you're getting the best combination 

of hardware designed for optimal performance 

with FreeNAS. Contact us today for a FREE Risk 
Elimination Consultation with one of our FreeNAS 
experts. Remember, every purchase directly supports 
the FreeNAS project so we can continue adding 
features and improvements to the software for years 
to come. And really - why would you buy a FreeNAS 
server from anyone else? 


vente 





FreeNAS 1U 

+ Intel* Xeon* Processor E3-1200v2 Family 

+ Up to 16TB of storage capacity 

+ 16GB ECC memory (upgradable to 32GB) 

+ 2x 10/100/1000 Gigabit Ethernet controllers 
+ Redundant power supply 


FreeNAS 2U 
+ 2xIntel* Xeon* Processors E5-2600v2 Family 
+ Up to 48TB of storage capacity 
+ 32GB ECC memory (upgradable to 128GB) 
+ 4x 1GbE Network interface (Onboard) - 
(Upgradable to 2 x 10 Gigabit Interface} 
+ Redundant Power Supply 





mei, Nelms og, Helnwel nscde logo and Xeon aretraidemaris of led Comoraton in HeU.S. mdoroter counvies 
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Secure VPNs with Gre and Strongs- 
wan for Small Business Networks 
with FreeBSD 


by Antonio Francesco Gentile 


Communication is a competitive advantage for any com- 
pany. With secure, reliable, and confidential communication 
it is possible to reduce costs, and improve the efficiency of 
production processes. In this article we will see how to im- 
plement them in FreeBSD. 





What you will learn... What you should know... 
In this paper we will learn to setup and man- Basic BSD Networking Setup, basic Network- 
age secure IPSEC over Gre Tunnels. ing structure knowledge, and basic Network 


security concepts. 





Startup 


A VPN (Virtual Private Network) is used when one needs to create a link between two or more pri- 
vate networks over a public network (like the Internet). Once the connection is established be- 
tween the two private networks, users will see the counterpart network in a completely transpar- 
ent way, as if they were physically connected to each other. But you have to keep in mind that the 
maximum connection speed between the two networks is defined by the public network, and not 
from the standard 100/1000 mbit LAN. Another very important characteristic of the VPN is to cre- 
ate a secure communication system; this, so you can be confident of security even in the case of 
transferring confidential and sensitive data. 
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FUPLICIF ieee as PUBLIC IP : 192.168.44.134 
LAN A 192.168 3.0/24 GRE IP : 10.255.255.2 
LAN IP : 192.168.5.0/24 
a een DMZ IP : 192.168.6.0/24 


CO GRE TUNNEL OVER IPSEC SO 


FreeBsd11-01 |psec over Gre 
Gawteway Firewall FreeBsd11-0? Ipsec over Gre 
Gawteway Firewall 


YP S 


Central Site Swith Branch Gffice Swith 











(mj 





Figure 1. A typical Small Business LAN to LAN Ipsec over Gre Scenario 


Summing up a VPN is "private" because it offers a nearly identical service to a private LAN, and 
"virtual" because in reality the data stream passes from the public network (Internet). It serves: 


1. For companies to connect peripheral headquarters without the need for dedicated connections. 
2. Two common users to share resources through the Internet. 

3. For an admin to administer a remote machine. 

4. To make a confidential shared network (WLAN or LAN of hub). 


All traffic between the two endpoints of the VPN is encapsulated in pre-tunnel. The tunnels can 
be on different levels of the ISO/OSI stack: 


1. IPSEC (a layer 3) 
2. PPTP (a layer 3) 
3. OpenVPN, I2tp, vtund, cipe, etc. (A layer 4) 


The implementation presented in this article will use the GRE tunnel over IPSEC Gateway Fire- 


wall FreeBSD 11. 


42 


MAGAZINE 


FreeBSD CORNER 








tunnel 








Figure 2. Gre Tunnel over the Internet. 


Our Scenario: 
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; HH) FreeBsb11amd64-fwgw 
=) Bp ipsecTest a 
} CeeOS Ot meyunger > Power on this virtual machine 
= EB ocserv a Edit virtual machine settings 
&) Gp ocserv2 
4 vm-101 
& [9 LinuxFwMultichains vy Devices 
| ED openwrt-ipsec i Memory 512 MB 
fH esxi6 ( Processors 1 
= a —— re Bdard Disk (SCSI) 120 GB 
5 Centos7-Omail 5) CD/DVD (IDE) — Auto detect 
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bsd11-Lan 

H] XpProFbsd11-01 

Sil] FreeBsb1 1amd64-fwaw 
psd11-La 

a FreeBsb1 1amd64-fwgw2 


virtual machine. 





y Virtual Machine Details 
State: Powered off 
Configuration file: D:\emulators\vmware\Fre...\FreeBsb11amd64-fwgw.vmx 
Hardware compatibility: Workstation 12.0 virtual machine 


aid MikrotikRouterOs-6 
Gl routeros-x86-6.0 
() Shared VMs 


























DAZ 





Figure 3. Vmware Workstation Setup of two remote networks connected over the Internet. 


The proposed scenario was implemented on a real gateway and tested on VMware architecture 
before it goes into production. Here is the network parameters used and the appropriate configu- 
ration file that was necessary to edit: 


(5-13 et Miele elongue 


WMelolonaylWerete(-1a welenalie 
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FIRST GATEWAY IPSEC SECOND GATEWAY IPSEC 
PUBLIC IP: 192.168.44.133 PUBLIC IP: 192.168.44.134 


GRE IP: 10.255.255.1 GRE IP: 10.255.255.2 
LAN IP: 192.168.3.0/24 LAN IP: 192.168.5.0/24 





An overlay VPN is a network of computers connected together that generate a given topology. 
The peculiarity of these networks is that they are not physical, but rather logical, and for this rea- 
son are made above other existing networks, in our case the INTERNET. The fundamental prereq- 
uisite for our project is that each node belonging to the overlay, or wishing to join this network, 
has a public IP address through which the management software may be placed in direct commu- 
nication. The first problem to be overcome is the logical interconnection between network nodes. 
In this regard, tunneling technologies come to the rescue. 


Tunneling is a technique that inserts a given protocol, in another protocol, at the same or higher 
level. The centerpiece of the whole project with this particular technique is to create virtual inter- 
faces on nodes that will make up the Overlay Network. Interconnection Point-to-Point between vir- 
tual interfaces on different nodes allows us to create the Overlay Network. GRE is the perfect can- 
didate as a tunneling protocol. 


GRE is already integrated in most Unix-like distributions and is very simple to setup. The decision 
to adopt this particular type of tunneling protocol is because GRE is able to encapsulate the in- 
side, also the broadcast and multicast traffic, used by common routing protocols. 


The nodes are physically interconnected thanks to the INTERNET network. 
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Security of data transmission is provided by IPSec in Transport Mode. This protocol is completely 
transparent to the application layer and is unable to implement encryption and authentication of 
IP connections. The Transport Mode it is made safe for only the IP packet payload. The IP header 
is removed and the payload is encrypted (ESP) and is applied to the security header (ESP/AH). 
Finally the original IP header is reapplied. To make this mechanism, configure the SPD (Security 
Policy Database) through which a SAD (Security Association Database) will then be generated. 


1. SP - Rule who should be treated by a specific SA. 
2. SA - it defines how to create a secure connection between two hosts. 


The connection is established and data of any type can be exchanged between all network nodes 
including routing tables to reach the new node. 


Pre-installation steps: 


If one prefers, instead of loading the appropriate modules in the loader.conf file it is possible to re- 
compile the kernel to support IPSEC and Gre. 


IPsec over GRE 
(GRE encapsulated packet is sent over the internet, routing updates and 
route information are exchanged in clear text. The interesting traffic 
defined for IPsec encryption is the actual payload which does not include 
the GRE traffic, so ONLY the underlying payload is encrypted.) 


Fosd11-01 GW Fbsd11-02 GW 


, 
1 me 
‘ . 
‘ : 
—_— — — / 
‘ . 
WapeeeeeeeeeeewaeeeTeeeTte eowacececclecccuscceceeucceces* 
e GRE IP 10.255.255.2/30 LAN 192.168.5.0/24 


‘ 
vo: j=, mmol 


x \ ~~. ‘ 
— — +t 
s. 2 


LAN 192.168.3.0/24 





GRE IP 10.255.255.1/30  ~**., 


‘ 
—_—— « 
’ 


ESP { 
(Data | (IPsec) } eS 


Figure 4. Operating diagram of gre over ipsec. 
Prepare FreeBSD kernel: 


The generic FreeBSD kernel does not come with IPsec support. So you will have to compile your 
own kernel. Fortunately, starting with FreeBSD 8, the NAT Traversal patch is included in the ker- 
nel sources, so you do not have to apply any patches yourself - if you need that feature. Let's 
start by customizing our new kernel configuration. 
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The standard naming convention for kernel configuration files is the name of the kernel in all caps 
and our new configuration will be called FBSD11IPSEC. Kernel configuration files live inside the 
/usr/src/sys/architecture/conf directory; the architecture used in this configuration will be AMD64. 


Change to the configuration directory. 


Copy the GENERIC kernel configuration file with our name “FBSD11IPSEC” and open it for edit- 
ing using your favorite text editor. 


You can find the FBSD11IPSEC configuration located here. Change the line starting with ident: 


To enable IPsec and gre one will need to add the following options to kernel configuration file: 


Now we will begin the kernel recompilation. Change back to the /usr/src directory and issue a 
make buildkernel utilizing your new configuration file. The build kernel will take some time de- 
pending on your hardware. Once the kernel recompilation has finished, it is time to begin the in- 
stall and reboot the system: 
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The server should now begin to shut down its currently running services, sync its disks, and re- 
boot into new kernel. One can check that new kernel config. is being used with the following 
command: 


syscel kern confttxt> | grep adent 





The output should be: 


ident [Ds SUD AMAL IME Sie 





One can verify that the kernel has IPsec support using the following command, which should print 
a list of the ipsec specific kernel state. 


fooun; Ssyscleie any |e grep me psec 


Installation of StrongSwan with FreeBSD Port/Package 


Historically, it made use of ipsec-tools suite to encapsu- 


Fbsd11-01 ba an ; , ” 
: Tun oe late the gre traffic with ipsec, with the “Racoon” service. 
ee els Request There are now more modern and flexible services avail- 
IPSec|Setting 
agra -. able on both Linux and FreeBSD, as StrongSwan will be 





IPSec is a used to make data on the tunnel protection. 





Figure 5. Operating diagram of gre over ipsec session. 


The easiest way to install strongSwan on FreeBSD is to use the security/strongswan port: 


cd /usr/ports/security/strongswan/ && make install clean 
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or to install the binary package with// Here this is just for the example, having a “optin” to enable 
our custom function: 


To get it to start at boot time, add this line to /etc/rc.conf: 


The proposed configurations make use of strongSwan version 5.4.0 Transversal with NAT func- 
tionality already active. For completeness, we will show the main configuration files of the two 
gateway firewall server. 











PP 192.168,44,134 - PuTTY = x 








In particular, one will show the two firewalling scripts, one realized through ipfw2, the native 
FreeBSD firewall, and the other with pf, the OpenBSD firewall. 
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@P 192.168.44.133 - PUTTY a o x 





In particular, one will show the two firewalling scripts, one realized through ipfw2, the native 
FreeBSD firewall, and the other with pf, the OpenBSD firewall. 


oP 192.168.44.1 

















roar = 
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sive etonobe ao: gew((O) eilestyels2 = ealsahone he) eerste oul) aimlcnenifelst<: Pac) Osa o)O he (A SNH Paap es 
detamltrouter— 197 16674472" 

cere Of@umle) Koike aliakone, ac var UG ioy cas IE Cal—netichs4.< wacle my aele yee): 0)e 

aimee hain Galle (i 1i( 4g shay lee Wa auhloyoer urge eak—gu)ti\okor <ty4 O1o)ny4wroln,are)i Oe 

SPs] aOl elylele lle Mad ane 

noNbbsrenclh teins aiaver: 

ibs ep verorele Ider “edapeyy 


dumpdev="AUTO" 


# DNS / DHCP Section 

iKever-e buglelevbualed w=iakeley a Mad eae 

claleqoree Roigtetodhe SW aaai oy! dhepd enabled? 

Glgle elo Laie cte/s Nop: command option(s) 
ditcpd congz—"/usr/local/ectc/dhiepd. cont” COnfaguracien file 
diced etraces— emily eng © Sie onaialoiey Neha nenmenei—7 (Gs) 


dhcepd: withumask="022" ae ISP wetial crete iepal. jivehed 


# Gateway Firewall setup 
Sieve vieinielellis dice 4 

## ipfw/natd 

avo cehy ciaheloi— aaa didi 


iavoneie a abaye, len eler ea 110by 


mabaderlags—)-f /Usr/ local/ectc/natd cont ® 
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eal ages Ie TSileved Kol sclsey | 
a ohicfeuive hy Feveneajene =) ohsiey/ IhevorWk (ciicie/ sein nieluers 


i ibigeneenl ab ere fepliays Fyn SY 


# MPDS 

ele Neiarel elke A Nah Sy) 

#mpd_ flags="-b -s mpd" 
eligje] oneOy Oia veil Eas Nee Ye 
NGS tpiwiltead= Veo. 


evel Mctchay kerche laa ddre is 


# TPSEC BLOCK 


SEIS Mr cla Malpelo vad ey 


# GRE BLOCK 
ie ARommsie oe ei eter orsi— vejig hy 


mete lay Galle pepe Owego, MO eA oie rast Ik AMO RZ Su AVA sisi. Sakseiioto< 510 AV Aiso na Sena ey 4 
tunnel 1£92.166.44.133 19216844. 134" 


SHE cede @@ibueisrs + sivhghaloalyy 


igo hey igs ebN diel aul a love re) ei Wy yee INCA raay Alone at 


Listing 2. The /boot/loader.conf file 
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Listing 3. The /usr/local/etc/ipsec.conf file 


‘Je Clore pelo) gin MmncdUraelelefon role uulilars(—1oMmole) ginakebnar-\omle gy enmald br 
# basic configuration 
config setup 

# strictcrlpolicy=yes 


# uniqueids = no 


SO Nolol welonaelsteiedneuers al=aes > 


Comma isd MiOi mb sci rao 


# peer IPs 

eT ett Zo ozo. 
livia wale li OMe alac) ayaa anne 
leftsubnet=192.168.3.0/24 


leftfirewall=yes 
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rightid=10.255.255.2 


rightsubnet=192.168.5.0/24 


# phase 1 parameters 
ike=aes128-shal-modp1536! 
ikelifetime=28800s 

# authentication 
authby=secret 

# phase 2 parameters 
esp=aes128-shal-modp1536! 
lifetime=3600s 

EV De=Eranspore 
leftprotoport=gre 
rightprotoport=gre 

# startup 

eiUe Ose chae 


keyingtries=%sforever 


Listing 4. The /usr/local/etc/ipfw.rules file 


Ho ain 


ae aliaswicdidl Marilee 
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# Macros 
fwomd=/sbin/ipfw 

Sob = Te eel, 

Dg arena ey 

seams clase hy eso 

SN 4s SS ta 

i monaey <i migjerass le Obe (0 LenC hale 


lntaliejlaley Sonatas. 1h IUO Zt aaloverenss ay 


# Vpn LAN Definitions 
lan net0="192.168.3.0/24" 


lan netl="192.168.5.0/24" 


dmz net="192.168.4.0/24" 


#ovpn lan="10.10.1.0/24" 


#Flushing firewall Rules 
#Sfwemd -f flush 


sy OREN oS Pe GLADE Sa 


#Allow SSH 


Snmiemicl vetelouOuhiG) sculivenis Gerele)Migicoiy cold: Geeyauie PAZ Mabey arene skis fs) sae bye) 
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# Allow Mpd 
Stwemd add 554 
Sr wemo sacl 555 


Ssiwemd add 0556 


# Allow L2tp 
SRmviel ie Mra Lolo MO hoo] 


Stwemd add 0556 


# Allow Ipsec Vpn 

Sfwcemd add 0559 allow 
Sfwcmd add 0560 allow 
Sfwemd add 0561 allow 


Sfwemd add 0562 allow 


# VPN Block 


# MPD Vpn Rulesets 


anys COR any Gee Spore wily 2s 
shane lay iiaey iene sWalny, 


lah Acre marol aly, 


any to me 1701 keep-state 


me to any 1701 keep-state 


Shs OPmbia(e) il wel aye WO Nasu al’s 
SUAle MENG O.I(ancbalia jelen solalyz 
bpenCcap rr Ome amy eo 


kolex. Waals wchalys wo) Mee 


Sfwemd add 0671 allow gre from any to any 


Shaenielicre lela Oem) Sarreullyonie gellilhy migey ii eligi: aeopaciuiva tavelesly (are 0 


Stwomd add 0680 allow all from any to any via ngs 
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NAT Section Start 


### Matching Internal/external IPs for Natting 
iii 
ii NAT Section End 


tEF 
#Final Rulesets 
Stiwemd add (65526. demy Lod) tcp irromeany torany in 


Siwemd- add 65534) deny Log all@from any to any ian 


Listing 5. The /usr/local/etc/ipsec.secret file 


OR tee Oo oe oo Chetek 2 


And we will explain all the configuration sets: 


On the Fbsd11 Gw2: 


Listing 6. The /etc/rc.conf file 


hostname="fbsd11" 
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Arenas ia’ 
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ane Crogpasheh Gomi a—Miali@ucise lone aks elo adh igls etielsle vaso nachos) nly 
Sis ialo! Mevgvelo NS Nees 

MQleHU Wek evel (oiavele Lice aldyoay 

ishejolelterakcile Ker agai. 


dumpdev="AUTO" 


# DNS DHCP BLOCK 
here 21D joindevenbliske Selalsiode —\ sdinen: 
ollavoyeyel yelinvellodier Nagas 


ellavel ore) gelests fs —iiaie 


Gitcpdecont= Us, local etc/dacod cont’ 


Gheed Piiraces— Vem Peenz: 
Giteiod SwaleMuiias k= 0.7 27) 
# Gateway Firewall setup 


SieVeeiely Rovereto ee! 
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# MPD5 
MN O16 Pu lat evel OME = ane alse S 
Pp Get leage= “be =coympa. 


SMaiOVONeLed. Cie well EAE NASM 


# IPSEC BLOCK 


Seo) sleve treba (—lelele ul and anon: 


# - Enabling the OpenBSD Firewall 


(ONE feige oS aginst 


pra nules—ustm/ local /ere/ pro comn” 


joen wedeenepse— lvl 
joel Mele (to iaysl elle aap 
pilogalogevile— var slog pr log” 


jonedere pac ene pens 


# Gre Block 
(ec oualistele shige. suatmeyeleve > Mieiiat0k' 


TieOmmigIgre0= tery LO 755.255. PLU e722). Netia Ss ia Do 00 a 
je obovate Uma Welles payee Roe Bs c-bedlee 0a Selle ronay aeleeo yo al 
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Listing 7. The /boot/loader.conf file 
Holeeliah jiicion Morte vdavey) 
lsnalase linen Wiersie aa Nimoy) 
ecial Mere sm Aneysie wd psi 


Vi abchale.<onbijelgl eilercle Sa gars 





Wate ace Oad= rio. 

povidiray load="YES™ 

df eee PlOed a Ero: | 

ale (ke shane che aweadie 

la felieakolos=. lheretel ais 

iii Aol oe mheyeNe ay eddies 

ele: Jeeyciehs vans 
#kqemu_load="YES" 

ieenae) lkeyevol= Madden 

# Load PF kernel modules 

jeu erie a Malays 

joie eres Aerio ars 

aesni load="YES” # [PSEC 
sh One eyete i valspese 

ah OVEN igyene, Mkcysuel ava Sy4 

al @f6 Wi erei ple eyel ey gtyS)i 

el oie Salighoue Ral obs 1% (eerolonmenuillye (elon Feleleicyehe ly 


net.inet.ip.forwarding=1 


lLalicieibaets; boise icine 4 
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Listing 8. The /usr/local/etc/ipsec.conf file 


HPLOSeCVCONE - SE LONG oWaAN, DP See) COME OUrabk1ensra le 
rip ole yslomhen a61e) dase utara eanen 4) 


config setup 
# strictcrlpolicy=yes 
# uniqueids = no 
keyexchange=ikevl 


dpdaction=restart 


us svolok: elojahmac eanolave: litanes 


(elonalay. tle) sve Minuit) ame stoll ah 1011! 


# peer IPs 
Itachi Operate ye aaeaa ne yan 

iReeeak olin O lease aaa 

leftsubnet=192.168.5.0/24 

leftfirewall=yes 
right=10.255.255.1 

sano lanemieikOlrga ao. Onder ab 


rightsubnet=192.168.3.0/24 
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# phase 1 parameters 
ike=aes128-shal-modp1536! 
ikelifetime=28800s 
# authentication 
authby=secret 
# phase 2 parameters 
esp=aes128-shal-modp1536! 
lifetime=3600s 
EVpe—EEanspore 
leftprotoport=gre 
rightprotoport=gre 
# startup 
ZU UND Cen echele 


keyingtries=%forever 


Listing 4. The /usr/local/etc/ipsec.secret file 
AOL Aro NS ae Arone mya ae, Needed, wd Galella Alara) 


Listing 5. The /usr/local/etc/ipfw.rules file 


###pf£.conf FreeBSD 11 


lan net = "{192.168.5.0/24}" 


ovpn net = "{192.168.58.0/24}" 
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# VPN IPSEC Rulse 
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192.168.44.134 - PuTTY 


rail 


Rs 
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Here, one use’s tcpdump to capture output over gre: 
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Conclusions 


GRE over IPSEC technology makes it possible to create VPN between different geographic loca- 
tions, it is available on all versions of BSD and Linux, and can be integrated into the kernel by 
adding the oppurtune options. It also offers a level of security and very high flexibility, allowing 
more end-to-end contemporary connections. It is therefore an excellent alternative to using dedi- 
cated hardware, expensive equipment provided by industry leading vendors such as Cisco Sys- 
tems, with which it is able to communicate by means of the correct configuration; additionally, a 
plus is that it is open source. The decision to implement gateway portal gre over ipsec, especially 
in small business environments simplifies the work of system administrators; thus, allowing them 
to get good performance, saving time, money and effort. 





| About the Author: 


| Antonio Francesco Gentile lives in Italy, Calabria, is a software and network engineer. He 
1 works for CNR, the National research center as network manager, with the “Culture Lab” 
| /ttp://culture.deis.unical.it Department of Telematics at University of Calabria, the com- § 
| puter science associations “Hacklab Cosenza” http://hacklab.cosenzainrete.it/ and 
} ‘Verde Binario” http:/www.verdebinario.org/ and is a freelance columnist for Italian maga- 
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Reusing the OpenBSD arc4random 
in Multithreaded User Space Pro- 
grams 


by Sudhi Herle 


Recently, a friend asked me for some help in speeding up 
crypto operations in her multi-threaded (pthreads) Linux pro- 
gram. She was using the standard /dev/urandom interface to 
generate random bytes. And, profiling showed that reading 
from /dev/urandom took a lot of time. | recalled that 
OpenBSD folks have a high quality userspace cryptographic 
random number generator called arc4random. But to the 
best of my knowledge, it was not generally available for use 
in any Linux program. And so, it seemed like an excellent 
weekend project to create a portable version of the 
OpenBSD arc4random(3) generator for use in multi-threaded 
programs. 





Before Starting: 
1. Make the latest version of OpenBSD arc4random(3) portable across Unixes and in par 
ticular Linux. 


2. _Noglobal state and no locks - so multi-threaded programs can work cleanly. 


3. _NoAPI change for programs; i.e., regardless of whether they were single-threaded or 
multi-threaded, the API should stay the same. 





The starting point was the arc4random(3) in OpenBSD libc. First was use of global variables for 
storing random state: 


As you can see, “rs” and “rsx” are both global variables. 


The rest of the arc4random.c source code refers to these global state variables; so, the first task 
was to change the functions to take an additional “state” argument. | collected all the state infor- 
mation needed into one struct: 
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/* arc4random state. */ 

SErUCE rand ystate 

{ 
size t rs_have; (oe i culaeh ten acreven che (oialel fond Jersy level 
Silvas Ie iGrs (OM DER NE fee Veiner: wakdidle simeistereie! 27 
isnliel ue iefes joalely [a Wibiae ASN Bia sar! 


Chachas crx imrsy (elnteNelatsl? = «9  (eIakeKelntch (elorMcene, imeie Actelintc lend 
keystream */ 


ae arte rs buf[ARC4R RSBUFSZ]; /* keystream blocks */ 
7 
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SMercliealen alsshllagicn avcelie: 
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return; 


if (rs == NULL) { 
ie 1 perce well Keleteuen (easy aaa csagcp.<) 


eleloneleal ey 





The change above was replicated to each of the internal functions. The modified version looks 
like so: 
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That leaves us with the external facing functions - arc4random_buf() and arc4random() and 
arc4random_uniform(). Each of these have a stable API used by userspace programs. The origi- 
nal functions used locks to protect the global variables. For reference, this is the original version 
from OpenBSD: 


We will be doing some clever things with pthreads to remove the locks. Before we do that, it’s use- 
ful to recap some functions in the POSIX pthreads API that will aid us in our quest. 


On a modern system that supports POSIX threads API (pthreads API), programs are allowed to 
create special slots to stash per-thread data. This is accomplished by calling the pthread_getspe- 
cific() API. Each piece of data needs to have a unique “key” associated with it. The per-thread 
data is set using the pthread_setspecific() API. However, before either of these APIs can be 
called, the key must first be created by calling pthread_key_create(). It is most critical that this 
key-creation is done only ONCE per key, per process. 


In order to guarantee that certain functions are only ever called once per process, pthread pro- 
vides a convenience function called pthread_once(). This function takes a function pointer as an 
argument and guarantees that that function is called exactly once. 
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Now, we have assembled most of the pieces to complete the puzzle: 


¢« We have removed global variables and added a new context variable for each of the arc4ran- 
dom internal functions 


¢ We know about the pthreads get_specific() and set_specific() APIs to manage per-thread con- 
text 


We are now ready to make arc4random() thread-aware and thread-safe. First, we create the func- 
tion that runs exactly once per process and creates the necessary key for pthread_getspecific() 
and pthread_setspecific(): 


We will return to pthread_atfork() later. Now, arc4random() becomes very simple: 
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The function “sget()” does a few things:: 

¢ Creates the key needed for pthread_get_specific() 

¢ Allocates state for the calling thread if needed and initializes random state if needed 
¢ Finally, returns the per thread state 


Its implementation is quite simple: 
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The first thing it does is setup for “screate()” to be called exactly once per process. When 
pthread_once() returns, it is guaranteed that screate() is called exactly once. So, when the call to 
pthread_getspecific() is made, either the per-thread random context is available or not. If this is 
the first time that sget() is called by this thread, then obviously, no per-thread context is available 
yet. So, it allocates memory, calls the required initialization code and finally, it sets the per-thread 
state by calling pthread_setspecific(). 


With the above changes, we have accomplished the following: 


¢ Eliminated the use of global variables for storing the random state; therefore, no need for locks 
to protect global variables 


¢ Created a per-thread state for the random generator - so each thread gets its own generator in- 
stance 


In order to ensure that the random generator state is not shared when a process forks, we need 
to detect when calls to fork() are made. Therefore, when we obtain the per-thread state, we check 
to see if the process has forked. We use two ways to detect this: 


1. An atomic counter that is incremented every time the process forks 
2. Current process pid compared against the pid in the random generator state 


Recall the screate() function above - where we saw a call to pthread_atfork(). This pthreads func- 
tion arranges for the supplied function to be called in the child process after fork(). Our implemen- 
tation of this callback function is quite simple. We increment a variable atomically. 


In the sget() function, we use this atomic variable and the current pid to detect if a fork has hap- 
pened: 
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In case the fork did happen, we reset the generator state and update the pid. 


The last remaining bit is to make a portable version of OpenBSD getentropy() call. This | accom- 
plished by making it a generic function that is implemented differently by each platform. On most 
Unix like platforms, that function is as simple keeping the file descriptor open and reading from 
/dev/random as needed. In the github repository below, a sample implementation of getentropy() 
for Linux is provided in the file posix_entropy.c. 


The source code for the portable generator and a simple benchmark is in github: 


To compile the example benchmark: 


The benchmark program is called “t_arc4rand”. It displays CPU cycles/per byte for each size. 
The “sysrand” column indicates the speed of reading from /dev/urandom. 


When run on a retina MacBook Pro 13” (2013) running OS X Yosemite: 
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As you can see, on OS X, the arc4random generator we just built runs almost 20x faster com- 
pared to reading from /dev/urandom! 


On Debian Linux (sid) x86_64 on a Core-i7 laptop running the Linux 4.5 kernel, | see: 


That’s a nice speedup compared to reading from /dev/urandom. 


The generator consumes approx 1.5kB of state per thread. 


The userspace port of the OpenBSD arc4random generator is useful in many projects that need a 
very high speed cryptographic quality random source. Its high performance is particularly benefi- 
cial to embedded systems. 


As a matter of good programming discipline, the design pattern outlined in the article is useful for 
anyone working with multi-threaded programs. If you think of having your internal library func- 


tions work on global variables, don't do that. 
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Instead, pass the necessary information as “context” in the first argument of the functions. At 
some point, if you really must use a global variable, then the technique above allows you to cre- 
ate per-thread state in a lockless manner. 


Source Code 


The source code for this is available in github: httos://github.com/opencofi/mt-arc4random 


| About the Author 











Sudhi Herle runs a large engineering and product management organization at Rhyth- 
mOne - an ad-tech company. In his free time, he works on interesting programs. His 
professional page is on LinkedIn (https://www.linkedin.com/in/sudhinerle) 


sudhi@herle.net 
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HOW TO install the XFCE 4.12 Desk- 
top on NetBSD 7 


by Curt McIntosh 





Before Starting: 
https://slice2.com/2016/01/30/howto-install-the-xfce-4- 1 2-desktop-on-netbsd-7/ 
This is an update to previous posts for NetBSD 6x: 

http.://slice2.com/2015/01/03/howto-install-the-xfce-4-desktop-on-netbsd-6-1-5/ 


http://slice2.com/2013/10/10/howto-install-the-xfce-4-desktop-on-netbsd-6-1-2/ 


For a lightweight functional desktop on NetBSD, install XFCE. As root, perform the follow- 
ing steps. This covers 32 and 64 bit x86 hardware. Since NetBSD essentially runs on eve- 
rything, simply adjust the repository path to your architecture from the list here: 
http://ftp.netosd.org/pub/pkgsrc/packages/NetBSD/ 





Setup your binary repository 


ela ie) YA vicued ell tenia icel serge aka 


IO bel alan bksway el day/s-naey/ je) 46 (ihian 1g orere ab] Me icai=lc ee ole ne 


vi /usr/pkg/etc/pkgin/repositories.cont 





and add path: 


For x64 
http.//ftp.netbsd.org/pub/pkgsrc/packages/NetBSD/amd64/7.0_2016Q1/All/ 


For x32 
http.//ftp.netbsd.org/pub/pkgsrc/packages/NetBSD/i386/7.0_2016Q17/All/ 
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This is for convenience and can be removed when done. 


Note: | do not Know why the encoded quote characters keep appearing after /ALL/ in the path 
statements below. It must be an html coding issue and | am not a developer. Just make sure that 
at the end of the path statement it ends with /7.0_2016Q1/ALL/” with no trailing characters. In 
other words, it should look like the paths depicted in step 1 above, only it must end in a ” charac- 
ter. 


For x64: 


For x32: 


calculating dependencies... done. Nothing to upgrade. 


121 packages to be installed (251M to download, 887M to install): 
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xfce4-xarchiver-0.5.4nb1l xfce4-wm-themes-4.10.0nbl xfce4-wm-4.12.3 
Reet ehmoderoOm il, Sine heed —eiMiintiace ile Otol 
xfce4-terminal-0.6.3nbl xfce4-settings-4.12.0nb1 xfce4-session-4.12.1 


xfce4-panel-4.12.0nbl xfce4-orage-4.12.1 xfce4-mousepad-0.4.0nb1 
xfce4-gtk2-engine-3.2.0nb1 xfce4-desktop-4.12.3 

Gig Osi! aah) oMmals a Ohad oat (rie o sal odie fil ec aA VA Un anro) 
elementary-xfce-icon-theme-0.6 xfce4-4.12.0nb2 





Note that you should not start X as root. Run the following for users on the system. For example, 
the user slice2 would be setup as: 
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Note: be patient, it may take a minute to load 
When prompted, select use default config. In the upper left, select Applications > Log out. 


This step is optional. Enter Y when asked to proceed ? [Y/n] for each app. 


Browsers and plugins: 


When done installing icedtea-web, run the three commands below to configure avahi. 


Install security apps, utils and shells: 
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GUI ftp/scp client: 


Office Suite and multimedia: 


You can launch liberoffice from Applications > Office, or enter the soffice command in an xterm. 
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About the Author: 


Curt Mcintosh is a Senior Infrastructure Engineer with Riptide Technology. A true jack of all 
trades, his experience spans multiple operatingsystems, applications and hardware plat- 
forms. Holding certifications such as CEH,SSCP, Linux+, Security+, and Cloud+, NCDA, 
VCP5-DCV, CCNA, MCSE, MCNE, hispresent focus is on securing systems from the OS 
to the network. To see more, visit the slice2.com blog. It's a blend of odd and obscure con- 
figuration steps that are either poorly documented or not documented at all by the vendor. 
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FreeBSD Flavors. Do We Need 
Them? Today...GhostBSD 


by George Bungarzescu 


A (not too deep) journey to GhostBSD - desktop and enter- 
prise options - compared to pure FreeBSD 


Years ago, | used Microsoft products, from operating systems to servers at my job. | always felt 
that, even if they have good products, the strategy to be closed source, even succesful from one 
point of view, can not cover the entire requirements of information world. Becoming a more 
skilled user, | have opened the doors to the wonderful world of Linux, BSD and other open source 
operating systems and products. 


Right now, | use at home for everyday tasks a Linux distro. | feel comfortable but | feel that some- 
how, in order to have a more user friendly environment, there is a continuous need for improve- 
ment. | will not go into why using Microsoft products can be the wrong strategy, but | will state that 
it is required to have alternatives, and, diversity is a good thing for everyone (just remember how 
some virus, ransomware and other malware have affected business these days). 


But wait, we are talking about Linux not FreeBSD, right? No. Actually, as you already know, the 
line between Linux and BSD is gray as they share a common history and open source ecosystem 
software. 


So let's talk a little bit about the history of Unix, BSD and Linux (the advanced users can skip the 
next part). 


Berkeley Software Distribution (BSD) is a generic name for operating systems that are close to 
the original UNIX design. Having in mind "a robust, general purpose, time-sharing computing plat- 
form which would not become obsolete every time the hardware change" in 1977 - many years 
before Linux was born, a team created the first version of a BSD-like system. 


According to Wikipedia "FreeBSD's roots go back to the University of California, Berkeley. The 
university acquired a UNIX source license from AT&T. Students of the university started to modify 
and improve the AT&T Unix and called this modified version Berkeley Unix or BSD, implementing 
features such asTCP/IP, virtual memory and the Unix File System. 
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The BSD project was founded in 1976 by Bill Joy. But since BSD contained code from AT&T 
Unix, all recipients had to get a license from AT&T first in order to use BSD. 


In June 1989, "Networking Release 1" or simply Net-1 — the first public version of BSD — was re- 
leased. After releasing Net-1, Keith Bostic, a developer of BSD, suggested replacing all AT&T 
code with freely-redistributable code under the original BSD license. Work on replacing AT&T 
code began and, after 18 months, much of the AT&T code was replaced. However, six files con- 
taining AT&T code remained in the kernel. The BSD developers decided to release the "Network- 
ing Release 2" without those six files. Net-2 was released in 1991". 


To be more accurate, the FreeBSD Project itself had its genesis in the early part of 1993, par- 
tially as an outgrowth of the Unofficial 386BSDPatchkit (source - FreeBSD manual). The first dis- 
tribution, FreeBSD 1.0 was released in December of 1993. There are many flavors derived from 
this old software, but four are most popular, FreeBSD, NetBSD, OpenBSD and Dragon Fly BSD, 
and they are used to build many of what we know as BSD distros. Some people argue that even 
Darwin (open source version of Mac OS X) shares a large portion of code with FreeBSD. Also, 
Microsoft Windows was inspired from and used for production purposes BSD operating systems 
(hosting for example earlier version of Hotmail and Microsoft website, the sockets design, tools, 
etc.). 


The point is that, before having the internet, DOS, CP/M, Microsoft Windows, Linux and graphical 
interfaces, we had a good, solid, and ahead-of-its-time operating system that had roots in the 
Unix software, and provided us with a base to build on. For example, "all modern operating sys- 
tems implement a version of the Berkeley or POSIX socket interface. It became the standard in- 
terface for connecting to the Internet. Even the Winsock implementation for MS Windows, devel- 
oped by unaffiliated developers, closely follows the standard"(Wikipedia). In fact, many other tech- 
nologies we have today, would never have been created without the existence of BSD-like operat- 
ing systems and distros. 


Having this understanding of BSD in generic terms, anyone can ask - what is the role of 
FreeBSD in the modern world and of the flavors we have? 


From this perspective, we must understand that most of BSD "distro" are server oriented. Even 
unknown to the public there are many servers, most of them used for hosting environments that 
are running on FreeBSD, NetBSD or OpenBSD. 


Besides the lack of marketing, there are interests in the open source world to keep alive BSD like 
systems and this interest came from, as a proof of the quality of code, a working group of a well 
known distribution, Debian. https://www.debian.org/ports/kFreeBSD-gnu/ 
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If on the server side things are clear, | personally found it a little bit hard to configure the graphical 
interface. But here comes the help of desktop oriented distributions, such as GhostBSD. 


Here are some thoughts from Eric Turgeon, Developement Leader at GhostBSD: 
[GB]: What was your reason to start GhostBSD project? 


[ET]: Before | started using FreeBSD, | was an Ubuntu user curious about real Unix and hacking 
software. | found Eric S Raymond's paper on How to be a hacker 
(http://www.catb.org/~esr/fags/hacker-howto.html). In that paper, he mentioned BSD Unix and | 
was curious about BSD. | did some research and installed FreeBSD and found that it's not really 
user friendly. | did some more research and | tried out PCBSD. | was a Gnome user and | did like 
what PCBSD was trying to achieve, but KDE was not my DE. From there, | wanted to start my 
own FreeBSD Distribution. This is when | started GhostBSD as a Gnome Alternative to PCBSD, 
which was KDE only at that time. 


[GB]: How closely related is GhostBSD to the BSD family of operating systems? 


[ET]: GhostBSD is basically FreeBSD with GTK DE (Desktop Environment), like MATE, XFCE, 
Cinnamon, and Gnome, all pre-configured and ready to use. Also, we are developing a GUI 
(Graphical User Interface) tool, like Networkmgr, Update Station, Software Station, GBI (Graphi- 
cal BSD Installer) and many more tools that are not found on FreeBSD. GhostBSD is using 
FreeBSD ports tree and pkg repository, there is no change to the default kernel. 


[GB]: What are, from your - main developer - point of view, the strengths of GhostBSD 
compared to FreeBSD? How about the enterprise class of products based on GhostBSD - 
virtualisation, clustering, etc.? 


[ET]: FreeBSD focuses more on a server OS and GhostBSD's focus is Desktop, but since 
GhostBSD's base system is FreeBSD, GhostBSD is capable to be used on a server. GhostBSD 
is clearly aimed at the home and office environment. GhostBSD is capable of doing day to day 
tasks, even gaming. 


[BG]: Funding such a project is hard. What part of GhostBSD attracts more funding? How 
about coding volunteers? 


[ET]: Yea, funding a project like GhostBSD is not easy task, GhostBSD is strictly funded by dona- 
tions, Adsense, by some partnerships and sponsors. Sponsors are basically helping us get more 
donations because we display a banner or logo linked to our sponsor's website that give them 
cheap advertising. Recently, we started a Patreon campaign, but so far it has not interested any- 
one. GhostBSD doesn’t generate enough money to finance full time development, so GhostBSD 
development is only in our spare time. Sometimes, some people contribute to documentation or 
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[BG]: Are there other proposed changes that will be specific to GhostBSD that will contrib- 
ute to future adoption? What are your future plans with GhostBSD? 


[ET]: There are a lot of features that we would like to see in GhostBSD, like integration of Tor by 
default ready to use, a tool to enable sshd and other things that are not setup by default. Also, | 
would like to start a GTK DE build for GhostBSD/FreeBSD. 


As a conclusion, BSD like systems are proved to be fast, stable, server and desktop ready. 
The only weakness we see is a not the lack of information or functionality but exactly having 
enough BSD flavors to help a user fit exactly their needs. Having more help from volunteers and 
a modern way of funding such project can also accelerate the development of specific functionali- 
ties and growth of BSD like systems adoption as day to day systems. 
GhostBSD especially offers an easy solution for desktop oriented users by providing a nice and 
easy to use installer and a familiar Gnome environment. 


FreeBSD clones? They are fast, stable and ready for any challenge, so please, use them! And 
spread the word. 
George 


P.S. Eric can be reached using ericturgeon @ ghostBSD.org or directly to ghostBSD.org 


ft About the Author: 

_ lam George, an open source enthusiast working as Senior IT Specialist. | have 
| more than 12 years experience using different operating systems and more than 
_ 10 years in software development. | am happy to help, in any way, the open 
source community and to promote open source based solutions. 

You can reach me at george.bungarzescu@gmail.com or via linkedin 
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Your ability to put time and effort in 
a consistent way Is the key to suc- 
cess. 


Fernando Rodriguez, Co-founder of KeepCoding 


by Marta Ziemianowicz, Marta Strzelec & Marta Sienicka 


[BSD Magazine]: Hello Fernando, how have you been doing? Can you introduce yourself 
to our readers? 


[Fernando Rodriguez]: My name is Fernando Rodriguez and I’m the co-founder of KeepCoding. 
I’m an iOS instructor myself and have trained teams around the world, ranging from medium 
sized companies in Bolivia to Facebook in Menlo Park. 


| have recently moved from Madrid, to the San Francisco Bay Area, where we recently created an 
American subsidiary of our company. 


[BSD Mag]:Can you tell us something about KeepCoding? 


[FR]: KeepCoding started four years ago in Madrid, Spain, with three students of my first iOS pro- 
gramming course. We now have 5000+ students around the world, mostly Europe and LATAM, 
with raving reviews. In the Spanish speaking market, we’re considered the “Ivy League” (although 
| prefer the “Justice League” ;-) ) of programming bootcamps. 


We have recently opened a subsidiary in the Silicon Valley and expect to start operating in the 
last quarter of this year. 


[BSD Mag]: What kind of students can join this school? Should they have any particular ex- 
perience? 


[FR]: Anyone with some programming experience can join. The web bootcamp is certainly more 
gentle than the Mobile one, but you don’t have to be a super talented individual. Quite the oppo- 
site, the common factor in our most successful students has always been something else: the will- 
ingness to sweat. Your ability to put time and effort in a consistent way is the key to success. Natu- 
ral talent is a plus, not a must. I’ve seen many extremely talented developers fail, precisely be- 


cause they lacked the willingness to sweat. BS 
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Most of our students have a full-time job and families. So did Tolstoy. He had 13 children, and yet 
War and Peace got written. Those who understand that, inevitably succeed. 


[BSD Mag]: Where did the idea of such a “school” come from? 


[FR]: It came from my own frustration while learning new concepts and tools. When learning a 
completely new technology, it’s very easy to dismiss the forest for paying too much attention to 
the trees. 


The amount of time you can save and the insights you can get when a more experienced devel- 
oper clearly points out to you what you need to know and why is outstanding. If this mentor is 
truly a great one, he will even find a way to make the process fun. 


This has always been our mission, become a Jedi Academy for programmers to learn nerdy stuff 
in a fun and efficient way! :-) 


[BSD Mag]: And you are a teacher yourself? What do you teach the most? What do you 
like to teach the most and what would you like to start teaching? 


[FR]: Yes, | am a teacher and | believe that’s a key part of our success. The owner of a Brazilian 
airline called TAM used to spend one hour a week working as a phone operator for his own com- 
pany, taking up customer calls like any other operator. This was his way of keeping in touch with 
his customers. | always thought this was a brilliant idea. 


Being a teacher is my way of keeping in touch with the students and the fellow instructors. If you 
want to build an education company, you must teach and learn on a daily basis. 


As for what | teach, | specialize in iOS development, both with Objective C and Swift. Right now, 
I’m working on a future course of “Full Stack Swift”: creating both the backend and iOS clients for 
Apps with Swift. That’s what | am looking forward to, as well as incorporating more functional pro- 
gramming concepts into our courses. 


[BSD Mag]: Which concept do you see as the most difficult to understand by your stu- 
dents? 


[FR]: Oddly enough, a pretty simple one: the delegate. This is a design pattern very common in 
Cocoa (the set of libraries used to develop for iOS and MacOS). Perhaps because it’s not fre- 
quently used in other environments, it takes some time to sink in. 


[BSD Mag]: And the concept that is in turn the most difficult to teach? 


[FR]: I'd say the same. The best way to approach concepts that are hard to teach is by using real 
world analogies. Once you find the right one, it all goes smoothly. And if it’s a nerdy analogy, so 
much the better. | tend to use a lot of Star Wars related examples: it works great and 


keeps the students entertained through the process. ;-) BS D 
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[BSD Mag]: But you are not only teaching in KeepCoding? 


[FR]: | have taught the Big Nerd Ranch’s Advanced iOS Bootcamps in Europe, LATAM and US. 
I’m also collaborating with Udacity in their iOS Nanodegree program here in the US. This allowed 
me to learn other teaching methodologies. | believe this is a good thing. 


[BSD Mag]: You are doing many things at the moment. How do you manage to be involved 
in sO many companies and projects? 


[FR]: With great difficulty and the invaluable assistance of Mr Steven Pressfield. ;-) Everything 
boils down to managing your time correctly and being able to save your “prime time” from interrup- 
tions. 


For example, | always plan my day the night before. If you start your day by planning, you already 
lost the battle: emails are popping up, notifications, phone calls, requests for help by co-workers, 
etc. To survive, you immediately enter a “crisis mode” and once all the fires have been put out, no 
real work has been done. 


My prime time is the morning, that’s when I’m the most productive for creative work: program- 
ming, designing course materials or writing articles. During that time, I’m 100% unplugged and fol- 
low my to do list by the book. 


During the afternoon, | plug all devices in, and deal with emails, Twitter, slack and whatever, but | 
know that I’ve done my work and now can devote some time for less important chores. Before the 
day ends, | plan the next day. When you're planning tomorrow’s tasks, you have some distance 
from those future needs, distance enough to clearly see what’s important and what’s urgent. Al- 
ways do what's important. 


If you’re struggling with too many things to do, by all means read “The War of Art” by Steven 
Pressfield. It’s a brilliant book that | recommend to all my students. 


[BSD Mag]: You have quoted Robert Frost on your Linkedin profile: “| am not a teacher, 
but an awakener.” What does it mean for you? 


[FR]: There’s a very interesting parable about the building of the pyramids: 


An old man sees 3 men working on the pyramids in ancient Egypt. He asks the first one, “What 
are you doing?” The first man replies, “I’m laying bricks can’t you see it?” 


He then asks the second one: “I’m building a wall with bricks.” 


He then asks the last one: “I’m building a monument so magnificent that for millennia, men will 


look to it with awe!” 
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[BSD Mag]: Do you think that on-site training programs have an advantage over online 
courses? 


[FR]: Yes, they do, but not for the students. An on-site training is vital for the instructor as it al- 
lows you to watch the students while they learn. This allows you to discover what are the bottle- 
necks, what parts are hard to grasp, what analogies work and which don’t. An on-site training is 
the debugger for an online one. You must go through it before releasing a new online course. 


We always do a few on-site sessions before moving ahead to an online version. 


[BSD Mag]: What does Engineering Master Bootcamp look like? Do you take people to the 
forest in the middle of nowhere, tell them to code for a whole night and hunt during the 
days? 


[FR]: | wish! :-) We actually work with what we call a blended model. We mix onsite activities 
where the students get to know each other as well as their instructors. It includes some fun stuff, 
but not hunting in the woods with a flint knife. | like the idea though, so we might consider it in the 
future! The rest of the classes are either online (with a schedule that must be followed and in real 
time with the instructor), or video lessons that the student can take at his own pace, but with dead- 
lines. 


It’s a very intensive program and not for the faint of heart. However, the “survivors” have been 
very satisfied and the only metric | was willing to consider, job placement and employability, has 
been a resounding success. 


[BSD Mag]: What is the most popular course? What do you think are upcoming trends, 
which skills will be the most desirable in the nearest future? 


[FR]: Right now, our bestselling products are the two tracks in our Startup Engineering Boot- 
camps. Both last eight months are are extremely practical and created for the real needs of the 
industry. Think of any successful digital product right now, say Facebook, Whatsapp, Evernote, 
Uber, AirBNB, Netflix, you name it. They all share a common architecture, with a backend (usu- 
ally a series of micro services working together), a REST API and several clients, web and mo- 
bile, usually. 


Backend 


BaaS 
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This is the basis for our bootcamps, the students learn how to build each component and how to 
integrate it within the whole system. They end up by creating a realistic market (think of a super 
local eBay) application MVP with all the components. This last part is done as a group, working 
with scrum. Some groups have even decided to apply their own business plan and create a 
startup. 


The main difference between both tracks, mobile and web, is the emphasis on each technology. 


| believe that both technologies, mobile and web, are safe bets for the foreseeable future. The 
success of one technology pulls the other: any successful mobile App needs a backend and vice 
versa. 


These are great times for being a developer! 


[BSD Mag]: What do you think about open source? Are you a fan of its communities, very 
devoted to DevOps? 


[FR]: I’m a big fan of Open Source and | consider it one of the greatest cultural advancements of 
the past century. | believe the great challenge now is to port this same concept to other fields of 
human knowledge. 


Some limited experiments have been carried out in the past and with great success. In 1959, Nils 
Bohlin, a Volvo engineer, “open sourced” the company’s most important invention ever: the three 
point seat belt. This brought more visibility to Volvo than any marketing campaign could have 
achieved and has saved innumerable lives since then. 


Hopefully, the XXI century will see the Open Source approach entering other areas, while preserv- 
ing the right for intellectual property. 
[BSD Mag]: What do you see as the biggest advantage of the open source approach? 


[FR]: One of the best ways to become a better engineer, software or not, is to become part of two 
types of teams. 


First, you should join on where you're the least knowledgeable member. Just by observing senior 
engineers solving problems and doing their work, you will gain insights and knowledge that no 
book will ever be able to teach you. It’s like watching two Grand Masters play while you’re learn- 
ing Chess. 


The other group you must join is one where you are the senior member. It will teach you to lead 
and manage. 


Thanks to the Open Source movement, we software engineers have enormous ease of doing this 
and tapping into the wisdom of the gods in our field. This is invaluable and | always 
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recommend my students become involved in some open source project in their field of interest. 


[BSD Mag]: What is the biggest challenge for your company at the moment? What is the 
biggest challenge for the industry? 


[FR]: We started with very small groups of students, devoting an enormous amount of attention 
to their needs. Almost all of our growth has been by word of mouth from delighted students. As 
personally satisfying as it was, this model was not scalable. 


We have improved on it and keep working on being able to provide a customized learning experi- 
ence of the greatest quality to as many students as possible at a reasonable cost. 


This is the great challenge of all the education industry. This is compounded here in the US with 
the enormous cost of traditional education, that can sometimes bankrupt students. As surprising 
as it may be to a European reader, many young Americans are fleeing the country because they 
cannot pay their student loans. 


The education industry is on a tipping point and is ripe for a complete disruption. The change will 
start here in the US, where the problems are more poignant, but it will change the global land- 
scape. My five year old son will probably never have a desktop computer or an internal combus- 
tion car. He will likely not have a college degree, nor need it. 


[BSD Mag]: Do you think it is better to have very specialized employees or people who can 
combine knowledge from many different branches of sciences and different skill sets, like 
business, marketing, coding and multimedia design, for example? 


[FR]: If we're talking about software development, the ability and willingness to learn is far more 
important than anything else. A very specialized employee will be useless to the company and to 
himself in four years if he doesn’t reinvent himself periodically. 


In a startup, a “specialized generalist’, someone who knows a lot about a specific topic but also 
understands other aspects of the business, is invaluable. 


If you’re a mobile developer, but also have a sound knowledge of backend development, if your 
sprint is done, you can help the backend team to cleanup their backlog. This makes you more 
valuable to the company. 


Having an understanding of the big picture is also invaluable for yourself, as it “increases the 
reach of your radar”: you can detect changes in the different technology trends and adapt quicker 
before becoming outdated. 


This is why, in our bootcamps, both the web and mobile ones, we don’t limit ourselves to teaching 
programming “hard skills”. We include design, agile and business literacy tracks. 
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[BSD Mag]: Do you have any advice for our readers? 


[FR]: The name of our company, KeepCoding, was inspired by a Johnny Walker ad | once saw in 
Beirut many years ago. It was created after the Israeli air force had destroyed many bridges in 
the south of Lebanon. It displayed Johnny, walking as usual, over a broken bridge. It said: “Keep 
Walking”. That's it. Nothing else. 


| loved it and felt it expressed my own beliefs: no matter what, keep walking. So no matter what 
the difficulties might be, keep walking. And if you are a developer, and programming is what you 
were born to do, by all means, Keep Coding! 


_ About Fernando: 


Fernando Rodriguez has 20 years of experience as a devel- 
oper and teacher. He is a co-founder of KeepCoding, a training 
company based in Madrid and Berkeley. He has trained 5000+ 
developers around the world, from Facebook to indie devs, 
both onsite and online. His online iOS course has been men- 
tioned on Financial Times, VentureBeat and Information Week. 


One of these years he will be able to devote full time to his real 
passion and talent: cooking! 
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Baroness Neville-Rolfe DBE CMG, in the recently released 
UK government document “Criminal Sanctions for Online 
Copyright Infringement” mandates a 10 year prison sen- 
tence for serious instances of copyright infringement. This 
intends to bring the penalties in line with those found guilty 
of copyright breaches in respect to physical goods. Will 
this amendment help to reduce piracy? 


by Rob Somerville 


At first glance, harmonizing the penalties for 
physical and digital copyright infringement 
would seem to be a logical step. Artists and 
business have every right to expect legal pro- 
tection from those that would exploit their in- 
vestment — time, emotional, mental, financial 
or otherwise — and make substantial profits off 
the back of their creativity. Defining where we 
draw the line between culture, fraud and theft, 
however, is a much more difficult concept. 
And this is where the whole issue of licensing 
and IP rights falls flat on its face. When does 
a creation become public domain? How much 
protection should we give to work X where 
this is based on cultural input from others? 
How original is an original work? 


The derision that was served on Apple Inc. 
concerning their lawsuit against Samsung 
about rounded corners is a classic case in 
point. This totally mendacious lawsuit is a 
good example of the real undercurrents that 
lie beneath the surface of any technology in- 
dustry. As riches are to be made, the vultures 


in the form of lawyers, accountants, PR de- 
partments and vested interests gather to- 
gether in an unholy coalition to annihilate any 
vestige of common sense, jurisprudence, natu- 
ral law and legal precedent. 


Having read the document signed by the Bar- 
oness, | personally don't think that her mo- 
tives are totally misplaced. Organized crime is 
an evil that needs to be addressed, and any- 
one who doesn't understand the level of pene- 
tration that society has suffered from needs 
their head examined. Until the pond life that 
steals somebody's identity is banged up in the 
big house for 10 years, maybe then | will have 
the heart to support a change in legislation 
that allows somebody to be prosecuted under 
the legal framework of a strict liability offense 
(e.g. no mens rea defense) and potentially re- 
ceive a sentence in some cases greater than 
those who have caused loss of life, disfigure- 
ment, or — shall we be cynical — have finan- 
cially raped society by “legal” means? 
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It is proposed that protection will be in place 
to isolate individual offenders from such a dra- 
conian sentence. Your average geek in his / 
her basement sharing files via torrent can ex- 
pect cease and desist notices, domain cancel- 
lation and, while not stated in the document, | 
suspect fines. To quote, the level of penalty 
should meet the scale of the crime. And 
therein lies the rub. 


Counterfeiting is as old as prostitution. The 
only judicial difference between the two profes- 
sions is the penalty if you are successful in 
your chosen vocation. The problem is that in 
the digital age, | can make a copy that is elec- 
tronically indistinguishable from the original at 
byte level at the touch of a mouse. So the 
whole argument about software piracy, theft, 
etc. is a cultural misnomer that those who 
have a vested interest would prefer to propa- 
gate. Theft is based on the assumption that 
you deprive the legal owner of use of X by 
spiriting it away. Breaking IP law — worst case 
— should be considered fraud. Yet the penal- 
ties for counterfeiting currency or historical art 
are traditionally far more severe than those for 
the pimp. 


In the final analysis, this proposed piece of leg- 
islation has the stench of vested interest 
about it. It is well known that there are counter- 
feit parts aplenty in the supply chain of critical 
industries — automotive, airline, IT and nu- 
clear. These components, should they fail, 
could cause serious injury or death. If some 
plutocrat is ripped off by buying a fake paint- 
ing, or the economy takes a hit by a craftsman 
who can recreate a viable currency note, this 
is fraud. An innocent person going about their 
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daily life dying in a car crash due to faulty 
parts, or a child suffering burns due to a coun- 
terfeit power supply or battery is a different 
matter entirely. 


The level of fraud and corruption in the West 
has reached proportions that would cause our 
great-grandparents to turn in their grave. Little 
focus, or indeed financial support, is given to 
agencies that understand the true scale of 
what is going on in our globalized, technology 
driven society. As a teenager, | abundantly re- 
corded music off the radio onto cassette tape. 
What concerns me more is the school col- 
league | knew at the time in my first Saturday 
job. A prolific thief, everything from expensive 
metal tapes to televisions were spirited away 
from the major high street store to be resold 
at a profit. Last time | heard, he was a detec- 
tive with the local police force. 


To quote “Penalty Fair? Study of criminal sanc- 
tions for copyright infringement available un- 
der the CDPA 1988”, the issue boils down to 
one simple question. Fundamentally, either on- 
line copyright offenses are capable of causing 
serious harm, or they are not. The difficulty 
arises right at the very top level of the legal 
tree — is this offense civil or criminal? 
Whereas in criminal law the guilty decision 
has to be made beyond all reasonable doubt, 
civil law is based on balance of probabilities. 
The inclusion of strict liability in this proposed 
legislation blurs an already muddy landscape. 


| have stated many times we have too much 
law, too many hooks upon which the necks of 
the relatively innocent may be hung. 
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It has come to the point that we will all go to with protecting corporate wealth than true jus- 
our graves law breakers. The key question is tice. 

whether or not we would prefer to be gov- 

erned by the letter rather than the spirit of the 

law. Looking at the corporate IT sector, the for- 

mer seems to be the trend. That is what you 

get for subscribing to the ethic of knowing the 

price of everything, but the value of naught. 


When it comes to judicial punishment, there is 
only one way to guarantee an offender does 
not repeat their offense ever again and that is 
to dispatch them — swiftly or otherwise — into 
the next life. Many countries have now re- 
pealed the death penalty, and offenses de- 
pending on their severity are met with every- 
thing from a verbal caution to a lengthy spell 
in prison. The punishment should fit the crime. 
One suspects that the zeal with which the Min- 
ister for Intellectual Property has approached 
the whole sentencing issue has more to do 
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